So, after all that fuss, which pollster (or meta-poll analyst) was the most accurate on the eve of the November 4th election?
I made the following chart to compare the election eve predictions of various major pollsters (and poll analysis sites) to see how their final projections compared with the actual numbers. The first column is how far off each outfit was in predicting the margin of victory (Obama minus McCain). The second column is Obama; the third McCain; the fourth is how far off each was for Obama; the fifth is how far off each was for McCain. The final row averages the numbers. The numbers are from 11/03/08, as reported around the web:
The chart and some comments follow after the jump...
Margin +/- | Obama | McCain | BO +/- | JM +/- | Outfit |
---|
-1.5 | 50 | 45 | -2.6 | -1.1 | Diageo/Hotline |
-0.5 | 50 | 44 | -2.6 | -2.1 | Battleground |
-0.5 | 51 | 45 | -1.6 | -1.1 | Research2000 |
-0.5 | 52 | 46 | -0.6 | -0.1 | Rasmussen |
-0.4 | 52.3 | 46.2 | -0.3 | +0.1 | FiveThirtyEight.com |
0 | 52.6 | 46.1 | 0 | 0 | *** ACTUAL RESULT *** |
+0.5 | 50 | 43 | -2.6 | -3.1 | FOX News |
+0.5 | 52 | 45 | -0.6 | -1.1 | Democracy Corps |
+0.5 | 53 | 46 | +0.4 | -0.1 | Ipsos/McClatchy |
+0.5 | 53 | 46 | +0.4 | -0.1 | Princeton El. Cons. |
+1.1 | 52 | 44.4 | -0.6 | -1.7 | Pollster.com |
+1.1 | 52.1 | 44.5 | -0.5 | -1.6 | Real Clear Politics |
+1.5 | 51 | 43 | -1.6 | -3.1 | WSJ/NBC |
+2.5 | 51 | 42 | -1.6 | -4.1 | CBS |
+2.5 | 53 | 44 | +0.4 | -2.1 | Marist |
+2.5 | 53 | 44 | +0.4 | -2.1 | Wash. Post/ABC |
+4.5 | 53 | 42 | +0.4 | -4.1 | USA Today/Gallup |
+4.9 | 54.1 | 42.7 | +1.5 | -3.4 | Zogby |
|
+1.1 | 52.1 | 44.5 | -0.54 | -1.64 | *** AVERAGE *** |
The prize for accuracy goes -- not surprisingly -- to Nate Silver of fivethirtyeight.com, who underestimated Obama's margin of victory by only four-tenths of one percent, coming within three-tenths of one percent of Obama's actual number, and within one-tenth of one percent of McCain's actual number.
As for the worst pollster, it depends which metric you pick.
Our friends at FOX News were tied for being the farthest off in terms of raw percentages for both Obama and McCain, allocating far too many voters to the "undecided" category on the night before the election. Their margin of victory call, however, was almost as close as Silver's.
My least favorite pollster, John Zogby, was the farthest off in terms of mis-estimating the margin of victory by +4.9%. USA Today/Gallup was also way off (+4.5%), getting Obama's number almost right, but badly underestimating McCain's number.
Overall, these outfits overestimated Obama's margin of victory by 1.1% on election eve, which is respectable enough. (Lest anyone see that phantom "Bradley Effect" in these numbers, note that the difference comes almost entirely from lowballing McCain, rather than underestimating Obama.)
However, in my opinion it is a lot easier to project a winner one day before an election than two weeks or a month (let alone three months) beforehand.
It is beyond my time and data-gathering abilities to compute, but I am hoping someone out there on the web will do a more thorough analysis of which pollster was most reliable on the most days over the period from when Obama sealed the nomination to election. But to do that computation would be complicated, since by election day there are no undecideds voting, but 3-4 months before there are many undecideds.
So to assess the accuracy of pollsters on, say, July 4th, 2008, would be a slightly subjective thing. Do you evaluate them based on the margin between the candidates on that date? Do you compare the raw numbers? And since early numbers are going to be skewed by undecideds, do you assess them by allocating undecideds in some fashion (50-50, or proportionally to the undecideds)?