The election is over, and the big winners were Democrats - and pollsters. Despite endless Republican hyperventilation over the accuracy and reliability of state polls, those polls proved to be extremely accurate in projecting the outcome of the presidential election.
Of course, people who had actually done the math knew that a relentless focus on aggregating state polls was the best way to project the outcome. This year we saw several different attempts to aggregate polls. Nate Silver's 538 is, of course, the best-known of all the aggregators, but was his formula the most accurate, in light of this year's actual results?
The answer is yes. According to my calculations, 538 remains the gold standard for presidential poll aggregation. Below the orange squiggle, I will detail how I came to this conclusion.
I'm no statistics whiz, so my evaluation did not include any complicated math. I simply took the final projected Obama-minus-Romney vote margins at nine different aggregating sites (some D-leaning, some R-leaning) for the 13 most competitive presidential states: AZ, CO, FL, IA, MI, NC, NH, NM, NV, OH, PA, VA, and WI. I then compared those projected margins to the actual results (as of this morning).
For each pollster-state combination, I calculated the miss (i.e., absolute value of projection-minus-result), and the bias (i.e., projection-minus-result).
Here's what I found (aggregators listed alphabetically):
Electoral-Vote.com - average miss 2.32 points; average bias R+1.16
Electoral-Projection-com - average miss 2.09 points; average bias R+1.64
FiveThirtyEight - average miss 1.61 points; average bias R+.50
Frontloading HQ - average miss 1.95 points; average bias R+.94
Gott/Colley Electoral Scoreboard - average miss 2.57 points; average bias R+1.89
HuffingtonPost Pollster - average miss 1.98 points; average bias +1.16
Princeton Election Consortium - average miss 1.99 points; average bias R+1.94
RealClearPolitics - average miss 2.52 points; average bias R+1.78
TalkingPointsMemo PollTracker - average miss 2.04 points; average bias R+1.22
As you can see, of these nine aggregators,
Nate Silver's 538 performed the best, both in terms of lowest average miss and in terms of lowest average bias.
Only 538, Josh Putnam at Frontloading HQ, Pollster.com, and Sam Wang's PEC projected these 13 states to an accuracy of less than 2 percentage points on average. 538's average miss was an impressive 1.61 percentage points.
Each of the aggregators' state projections, on average, were biased toward Romney. But only 538 and Frontloading HQ had an average bias of under one point. 538's average bias was only half a point.
That is, 538 was the most accurate aggregator; and, when it was inaccurate, it wasn't biased - it was just about as likely to be biased toward Obama as it was to be biased toward Romney.
If I had to venture a guess as to why 538 performed so well I would point to the one thing that separates it from the crowd - individual poll weighting. Other aggregators employ a cut-off date when determining which polls to aggregate; Nate uses a complex (and secret) formula that takes into account pollster house effect, sample size, date, and other factors. As far as I know, he is the only aggregator to do this. Whatever Nate's secret sauce is, it worked better than any other poll aggregator out there in 2012.
Well done, Nate. You deserve a well-earned vacation!