Back on November 1st, when posting my final Senate and Governor Snapshots, I wrote:
We won’t have final vote totals everywhere until mid-December (and possibly later). When that time comes, I will compare the accuracy of my methodology to Five Thirty Eight, Pollster.com, and Real Clear Politics. But really, all of our predictions are pretty much the same. The difference in the average accuracy of our predictions will only be a few tenths of a percent.
Now, the time has come to do just that. And, as predicted, the average difference in accuracy was only a few tenths of a percent. But, there were still differences nonetheless.
To make this comparison, for the 49 campaigns I projected, I used the difference between the final actual vote percentage margin on DaveLeip’s site (see here, numbers as of this morning) and the final projected percentage margin for myself (see here), fivethirtyeight (see Senate here and Governor here), Pollster.com (see here), and Real Clear Politics (see Senate here and Governor here). Some quick notes on how this was done:
- If a forecaster’s final projection was the Democratic candidate winning by 10.0, and the final margin was a Democratic win of 12.63%, then that forecaster is given an error of 2.63 for that campaign.
- The total error for all 49 campaigns is added up, and both the mean and median are determined.
- The 49 campaigns I used in my forecast were the 49 Senate and Governor campaigns projected to be within 18.5% or less on the night before the election.
- Both 538 and I projected results for all 49 of those campaigns, so when those two methodologies are compared all 49 campaigns are used. Pollster.com did not do a projection for the Vermont Governor’s campaign, so the three-way comparison between Pollster.com, 538 and myself, only 48 campaigns are used. Real Clear Politics did not do a projection for the Alaska Senate campaign, Alabama Governor, or Oklahoma Governor, so when all four methodologies are compared, only 45 campaigns are used.
- The master comparison table can be downloaded as an Excel file here. It is possible made some errors, but I worked hard all weekend to make sure that I did not. Any existing errors are accidental.
Here are the results:
In the two-way comparison between 538 and my simple, 25-day day polling mean, I slightly outperform 538 on the mean. However, 538 slightly performs me on the median.
In the three-way comparison that includes Pollster.com, I narrowly edge out 538 on the mean, with Pollster.com a little bit behind. Once again, 538 does better than me on the median, and Pollster.com is a bit further back on that measure.
In the four-way comparison, 538 does better in both measurements, and I am in second in both. Real Clear Politics narrowly edges out Pollster.com on the mean, and Pollster.com is ahead of Real Clear Politics on the median.
Now, the median is a bit more important than the mean, since individual instances of catastrophic error can skew the results of the mean. Also, the reason I am ahead of 538 on the mean in the first and second comparisons was due to the extremely difficult to poll and predict campaign for Senate in Alaska. In that campaign, I also actually closely predicted the final results (Murkowski +4.02, while I had Murkowski +2.9), while Nate had one of his worst predictions (Miller +6.1). However, I had begged for a mulligan on that campaign for quite a while, so it doesn’t feel quite right to use it as a way to declare overall victory.
With all that in mind, this produces the following ranking of the four forecasting methodologies for 2010:
- 538
- Bowers
- Pollster.com
- Real Clear Politics
I should note that a forecaster friend of mine is currently doing a wider analysis between Pollster.com and 538, looking at more than just the 49 campaigns I projected in my final Snapshots. He has indicated to me that the preliminary results show Pollster.com performing substantially better than 538. I will leave it to him to post the results when he is finished.
Further, even though I haven’t run the numbers, I’m pretty sure that I would have had the most accurate forecast if I had just removed all Rasmussen polls from my projection (Rasmussen did terribly in 2010). I was tempted to do this at several points during the campaign, most notably here, but I just didn’t have the guts. In this case, it would seem that no guts means no glory. I have no one to blame but myself for that one--the opportunity was there, and I passed.
Still, it’s striking how incredibly close my brutally simple, even borderline stupid, methodology came to the far more complex methodology of 538. Just taking the average of all the polls (expect Zogby interactive polls) conducted over the last 25 days gets you just as close to the final result as anything else, at least in statewide campaigns. Anyone can replicate my methodology, too. So, if nothing else, the 25-day polling mean is a small victory for transparency and amateurism.