NPI Ratings and Rankings Guide

A. Guide to Ratings:

Why these three criteria?
Each NPI review includes Ratings of the piece being reviewed with respect to three criteria. These criteria were selected after consideration of what are the virtuous attributes that can be assigned to written punditry from a non-partisan standpoint, that is, independently of whether one shares the political views of the pundit.

Three criteria: Ratings from 0 to 10
The three NPI Rating criteria follow. They are accompanied by a brief description and illustrative examples of the kinds of reactions that would make a Reviewer lean more towards awarding either a good or a bad Rating. Each criterion Rating can take a value between 0 and 10.

Criterion 1: Insight
You don’t have to agree with the pundit’s argument in order to recognize that it is insightful. Nor do you have to share the pundit’s political views. Insight is like charisma; it’s not predicated on agreeing with the person and you know it when you see it.
Good: “I don’t like his conclusion but this makes so much sense!” “A very interesting and novel take on the issue.”
Bad: “Did she just copy-paste the transcript of last night’s debate? Not a single original thought.”

Criterion 2: Style
Image matters in politics and it matters in punditry too. A pundit’s writing style is what shapes his image.
Good: “His imagery is moving.” “Her diction is well-chosen.” “Has a nice flow.” “Witty tone.” “Clever title.”
Bad: “Her sentences are too long!” “These analogies don’t really work.” “She’s a bit dry.”

Criterion 3: Evidence
Does the piece make reference to evidence? Is the evidence interesting? Relevant? Did it likely require unusually intensive or perhaps even original research? Equally importantly, is it used well to support the piece’s thesis?
Good: “That’s exactly the kind of quote that was needed here.” “The numbers make so much sense analyzed that way.” “Why hasn’t anyone else discussed this polling data?”
Bad: “These quotes have been used by everyone.” “The evidence presented is interesting. Too bad it is completely irrelevant to the thesis.” “The piece seems to lack a clear thesis so it isn’t obvious whether the evidence presented is relevant to it.”

Ratings on a strict curve so that they are meaningful
The way we think about each Rating is that if a piece appears to the Reviewer to be about average with respect to a certain criterion, the Reviewer will give it a 5 for that criterion. A distinctly positive reaction will tempt him to move up from 5, while a distinctly negative one will tempt him to move down from 5. Indeed, our goal is to award Ratings so that, over time, all the Ratings by a given Reviewer for a given criterion are distributed on a normal curve, with 5 being the mean, median and mode Rating. Thus, for each criterion, you should expect to see Ratings of 4 and 6 with roughly equal frequency, Ratings of 3 and 7 with roughly equal frequency, and so on.

A review’s Overall Rating
A review’s Overall Rating of a Piece is the sum of its three criteria Ratings and can range from 0 to 30. Accordingly, our aim is that the Overall Ratings will also be distributed on a normal curve, with 15 being as close as possible to the mean, median and mode Overall Rating.

Thus, Overall Ratings above 27 or below 3 should thus almost never appear. A pundit that writes a piece that receives an Overall Rating of 28 or above is considered to have reached The Holy Grail of Punditry, while one that writes a piece that receives an Overall Rating of 2 or below is considered to have descended to The Pits of Punditry. Such classifications occur extremely rarely.

“A review’s Overall Rating of a Piece” versus the “NPI Overall Rating for a Piece”
You will notice when browsing through our reviewed pieces that we present the Ratings awarded by the original review of the piece and then we also present, at the bottom of the reviewed piece, its NPI Overall Rating. The way to think about these is that the NPI Overall Rating for a Piece is the official Overall Rating that is based on the reviews of the piece that are displayed right above it. You will also notice that for the vast majority of reviewed pieces, there is only one review – the original review – and that as a result the NPI Overall Rating is equal to the Overall Rating of the original review of the piece. But, alas, there are certain cases in which there will be more than one review written of a given piece…

Peer review: Additional reviews of a piece required if its original review awards an extreme rating
To ensure the credibility and meaningfulness of these official NPI Overall Ratings, any original review of a piece that awards an Overall Rating that is below 3 or above 27 will automatically trigger a peer review, i.e., will trigger additional reviews of the piece. The number of additional reviews of the piece that are required will depend on the extremity of the original review’s Overall Rating:

Original review gives 2 or 28 – a minimum of 1 additional review
Original review gives 1 or 29 – a minimum of 2 additional reviews
Original review gives 0 or 30 – a minimum of 3 additional reviews

Additional reviews do not offer a “Thesis summary” or a “Punditry of punditry” section; they only offer an independent set of Ratings.

A peer review-triggering original review will be displayed within the layout of the reviewed piece as normal while the additional reviews are quickly completed by other Reviewers, the only difference being that the NPI Overall Rating for the Piece will display “Pending” instead of just showing the original review’s Overall Rating. Once all additional required reviews have been published right below the original review, the NPI Overall Rating for the Piece will change from “Pending” to a number. But what will that number be???

Determining the NPI Overall Rating for a Piece in the case of more than one reviews
Once all required additional reviews are published, NPI calculates the arithmetic average of the Ratings of all the reviews (original and additional) for each of the three criteria and then sums these three averages up to the nearest integer to arrive at the peer-review Overall Rating for the piece. If this peer-review Overall Rating is more extreme than the original review’s Overall Rating, then the original review’s Overall Rating for the piece is confirmed as the piece’s NPI Overall Rating. If this peer-review Overall Rating is less extreme than the original review’s Overall Rating, then the peer-review Overall Rating becomes the piece’s NPI Overall Rating. Notice that the NPI Overall Rating will never be more extreme than the original review’s Overall Rating.

Only if and when a piece that was awarded an original-review Overall Rating of 28, 29 or 30 has emerged from the triggered peer-review process with an official NPI Overall Rating that is 28 or above will NPI declare that the pundit in question has reached The Holy Grail of Punditry. The opposite holds for pieces originally awarded a 0, 1 or 2 and the declaration that a pundit has descended to The Pits of Punditry.

B. Guide to NPI Rankings:
Our NPI Rankings page gives NPI’s readers the opportunity to see what all our reviews and Ratings can tell us about the pundits that write these pieces and about the publications that publish them, both compared to each other and over time. Note that a piece only goes “live” (i.e., factors into the Rankings tabulation) when its NPI Overall Rating has been determined, since Rankings are based on NPI Overall Ratings, not Overall Ratings of individual reviews of pieces. In other words, if an original review has awarded an Overall Rating that required additional reviews, until those additional reviews are completed, that piece will not be factored into the Ranking of the pundit that wrote it or the publication that published it.

All Rights Reserved © 2008 No Punditry Intended LLC