In this paper, we report on the first-ever test of the accuracy of predictions in the media. We sampled the predictions of 26 individuals who wrote columns in major newspapers and/or appeared on the three major Sunday television news shows (Face the Nation, Meet the Press, and This Week) over a 16 month period from September 2007 to December 2008. Collectively, we called these pundits and politicians “prognosticators.” We evaluated each of the 472 predictions we recorded, testing its accuracy.
So reads the introduction to the executive summary of An Analysis of the Accuracy of Forecasts in the Political Media by Hamilton College students Holly Donaldson '11, Russell Doubleday '11, Scott Hefferman '11, Evan Klondar '11 and Kate Tummarello '11.
Guess what? Liberal prognosticators are more accurate than conservative ones. Having a law degree is negatively correlated with accuracy, which in a related study was been shown to be an enormous advantage when arguing before a jury. [Okay, that second part is a joke. No one has ever bothered to do the study.] The average pundit is as accurate as a coin flip, no matter what the subject under discussion.
To obtain our final prognosticator sample, we first generated a sample of 22 print media columnists and 36 TV prognosticators, based on who was most widely syndicated and who appeared on the network Sunday morning talk shows more than 5 times within our evaluation period. From our full sample of 58, we then randomly selected 25 prognosticators to create our final sample. Later, we added George Will as our final (26th) prognosticator, due to his enormous presence in the media scene.
And the winner is . . .
Oh damn, I already gave it away.
The most accurate prognosticator was Paul Krugman of The New York Times and Princeton University, followed by Maureen Dowd, another columnist for the Times, and former Pennsylvania governor Ed Rendell. The worst prognosticator was Cal Thomas, a syndicated columnist. South Carolina Senator Lindsay Graham was next worst, and Michigan Senator Carl Levin was third worst.
Each pundit earned a Pundit Value Score (PVS) based on his predictive accuracy. (The intangible negative value of pundits was not considered, but I'll mention that below.) While the group as a whole performed the same as chance, a few stood out, most broke even, and a few were just dead wrong.
The Good (listen to them, but get a second opinion): Paul Krugman, Maureen Dowd, Ed Rendell, Chuck Schumer, Nancy Pelosi, Kathleen Parker, David Brooks, Eugene Robinson, and Hank Paulson.
The Bad [PVS slightly better than Aunt Bessie's rheumatism): Howard Wolfson, Mike Huckabee, Newt Gingrich, John Kerry, Bob Herbert, Andrea Mitchell, Thomas Friedman, David Broder, Clarence Page, Nicholas Kristof, and Hillary Clinton.
The Ugly (using a Ouija Board would IMPROVE their performance): George Will, Sam Donaldson, Fightin' Joe Lieberman, Carl Levin, Lindsey Graham, and Cal Thomas.
Conditional predictions, as opposed to absolute statements that a thing will or will not occur, were less likely to be accurate. Liberals were more accurate than conservatives even when predictions of the outcome of political races were removed from the data.
And the loser is . . .
The voting public
[Emphasis gleefully added]
First, we have discovered that 9 of the analyzed prognosticators are significantly better than a coin flip. Two are significantly worse, and the remainder are not significantly different, in a statistical sense, than a coin flip. This means that the average prognosticator is no better than a coin flip in terms of the accuracy of his or her predictions.
A final important implication is that we did not discover that certain types of predictions tended to be more or less accurate than others. For example, we did not see that economic predictions were more accurate than healthcare predictions. This suggests that prognosticators on the whole have no unique expertise in any area—even on political predictions, like the Presidential or party primary elections.
This is interesting:
... the overwhelming majority of demographic factors have no bearing on a prognosticator’s accuracy. Gender, race, and age [except with respect to extreme language] are all irrelevant, as are most career path choices, such as becoming a journalist.
When listening to current political advisers, keep in mind that they are more likely to make predictions using positive language. Similarly, it is important to remember that professional journalists and male prognosticators are more likely to make predictions using negative language. Younger prognosticators, politicians, and journalists with more experience are more likely to predict using extreme language.
If watching the pundits makes you feel riled up and quite confident you are right, there may be a reason:
This tendency suggests that society at large should not look to prognosticators for nuance. While they may end up vindicated by a correct prediction or shamed by a misguided one, they will rarely reflect the uncertainty that exists in the real world....
Very few prognosticators were keen to use this nuanced language [of uncertainty and unpredictability] as our PredEXTREME regression shows. Politicians were much more likely to use extreme language, and journalists with more experience were also much more likely to as well.
Or perhaps--could it be?--that it's all just hot air:
... the prognosticators with much larger samples also tended to have Prognosticator Value Scores close to zero. This could be proof that PVS fall closer to zero as the number of predictions rise. George Will had the highest number of predictions in our sample (61) and has a PVS of -0.04, which is extremely close to zero.
Okay, okay, you want some nuance? Look, our side won fair and square, but the nitpickers may want to consider this:
Our findings seem to agree with Tetlock’s research. Some studies suggest that conservatives have more rigid ideologies (Jost et al., 2003) In other words, they would be considered “hedgehogs.” Similarly, lawyers are taught to argue one side with a single analytical method; they, too, would be “hedgehogs” under Tetlock’s model. While not all liberals are foxes and not all conservatives are hedgehogs, these trends may be informative in explaining why our results are as they are. It may be that conservatives are inherently disadvantaged as prognosticators due to their ideological “hedgehoginess.”
When is a prediction not a prediction?
When it's an attack or a veiled treat.
This brings me to a little commentary of my own. Publicly predicting the future with confidence may serve a purpose other than accuracy. Anyone who follows political races knows that making people believe you are going to win is highly correlated with winning. Other functions can be served as well. Consider the fairly common dailykos prediction of the sort: "Such and such an idiot or group of idiots will be here with their misguided, [insert ad hominem] comments in 4 . . . 3 . . . 2 . . . 1." This is not a prediction so much as a preemptive strike. It is an argument which isn't really an argument. Consider, "If Obama were to pull out of Iraq tomorrow, the economy would crash." This isn't a prediction so much as a scare tactic. Certainly the media is saturated with such self-serving "predictions," and just as certainly, they have an impact irrespective of their accuracy.
I'll end with an excerpt from a book that is currently opening my eyes wide with respect to the nature of much the discourse in the U.S. today. The author is Stjepan Meštrović, sociology professor at Texas A & M. The book is Postemotional Society. After citing numerous examples of his contention that "contemporary Western societies are entering a new phase of development in which synthetic, quasi-emotions become the basis for widespread manipulation by self, others, and the culture industry as a whole," Meštrović summarizes:
[Emphasis added]
For some readers, the immediate tendency will be to dismiss these and other examples as just old-fashioned instances of scapegoating, demagoguery, and propaganda. But I would suggest that even if the phenomena used for illustration above are somewhat similar to traditional phenomena, several things are new: (1) The camera is ubiquitous and therefore makes all these emotional phenomena potentially staged, artificial, and second-hand. (2) What David Riesman called other-directed “inside-dopesterism” is also a ubiquitous mode of consuming information nowadays, thereby transforming most persons into voyeuristic consumers of these second-hand emotions. (3) Even twenty years ago, there were no full-time, professional “opinion-makers” or “newsmakers.” People were not sure what opinions they held or ought to hold, and newsmakers were recognized much after the fact. By contrast, the 1990’s is the age of live newsmaking created by persons designed by as newsmakers whose opinions are immediately refracted by designated opinion-makers and whose impact is gauged almost as quickly by opinion polls, which are broadcast back to the inside-dopesters. It is difficult to conceive the possibility of feeling a genuine, spontaneous emotion in such a social structure. (4) Finally, in all of the illustrations that have been examined, and others that the reader can supply, one cannot help suspecting a climate of affected, feigned emotion, as if it were rehearsed and planned ahead of time.
Updated by geomoo at Tue May 03, 2011 at 09:06 PM EDT
Some commenters are talking liberal/conservative. According to the kids at Hamilton College, Dowd is the best darn liberal of them all. This made me think of the conspicuous absence of the infamous professional left, those extreme radicals who always threaten to screw everything up with their commentary. Apparently so far, they are all tucked safely away on the toobz.
STOP THE PRESSES! We have a prognisticator in our midst. From a comment by riverview:
I'm not a professional pundit, but... (1+ / 0-)
Over the last 30 years or so here are the things I ranted about well before the results were in:
+ Unionbusting would result in lower wages for the middle class.
+Privitization would end up costing more in taxes.
+ Free trade would lead to a decline in jobs for Americans.
+ Deregulation of the financial sector would cause severe problems for the economy.
+ The charter school movement would lead to the demise of public school education.
+ Tax cuts for the rich would lead to a widening gap between the rich and everyone else.
+ Ending the "Fairness Doctrine" for broadcast media would lead to greater influence for conservatives on the airwaves.
+ The War on Terror would create unsustainable costs for the U.S. taxpayers.
Hmmm...wonder how I'd score?
How does (s)he do it?