With ‘Super Tuesday’ behind us, we have a chance to take a look at how all the pollsters performed. Before throwing up numbers, I’d like to outline my approach to evaluating pollsters' performance.
.
#1: QUANTIFYING PERFORMANCE
How do we represent how ‘well’ a pollster performed? I think most of us will agree that the quantity of prime importance is the ability to predict a winner. However, any polling will involve a modest amount error and I don’t want to penalize pollsters who correctly project a close race, but get the winner wrong.
This hasn’t happened yet because so few of the races have been truly close (CT, MO, NM). To make my point, consider the hypothetical where Clinton, instead of Obama, wins Missouri by a razor thin margin. In this case SUSA (C 54, O 43) would have gotten the victor right, but would have missed on the margin of victory by a couple of touchdowns. Playing with this scenario a little further, Zogby (C 42, O 45) would have predicted a wrong victor, but would have correctly predicted a close battle. In such a scenario, I am greatly tempted to credit Zogby with having gotten this hypothetical race ‘right’.
Therefore, instead of crediting pollsters for projecting a winner, I have given them credit based on how close they got to the final margin of victory.
Besides the margin of victory, I am also interested in how close a pollster got to a candidate’s vote share. A 55-45 poll is not the same as a 45-35 poll. The former predicts solidified race, while the projects a more fluid one with greater room for movement. They are not the same and should not be treated as such.A poll that 'punts' on figuring out where undecideds go should be penalized.
It is understandable for pollsters to find a large number of undecided voters a couple of weeks out from an election. However, there is no excuse for having a large number of undecided voters in the waning days of a campaign. For this reason, I have only evaluated polls published within four days of an election.
Proceeding with the approach outlined above, I evaluated polls against each candidate's actual vote share and the margin of victory. I decided to double weight a pollster’s margin of victory projection, believing it to be a more important quantity than each candidate’s vote share.
To quantify a pollster’s performance, I’m going to butcher the concept of a root mean squared error (standard deviation).
Pollster Error = Sqrt [0.25 x ((Clinton Vote Error)^2 + (Obama Vote Error)^2 + (Margin Error)^2 + (Margin Error)^2 )]
While I do not claim this as a necessarily sound analytical approach (weighting the error of variables, etc...), this number does give you an idea what the average error across each the following quantities was:
(1) Obama’s vote share
(2) Clinton’s vote share
(3) Margin of victory (x2)
.
#2 ACCOUNTING FOR GEOGRAPHIC CHALLENGES
No one needs a degree in rocket science to know that polling Georgia is a lot difference than polling, say, Massachusetts. A pollster that absolutely nails the vote in the northeast could easily fall flat on its face when polling the south or the midwest (as SUSA did).
There were 12 ‘Super Tuesday’ states that both held primaries, and had sufficient polling information in the days just prior to the election. These states were – MA, CT, NY, NJ, IL, CA, MO, TN, OK, GA, AL and AZ.
Based on demographics, these states fall into four categories – Democratic Strongholds*, Lower Plains/Upper South, Deep South (high African American population) and Southwest.
* - Although California and Illinois are geographically distant from New York, New Jersey and Massachusetts, demographically they are close cousins. Democratic Strongholds are characterized by large metropolitan areas, ethnic diversity, a strong union presence and a greater density of higher education institutions. These states are reliably Democratic and economically prosperous.
.
THE RESULTS:
SUMMARY
Poll Error Averaged Across Political Regions
Democratic Strongholds:
Survery USA = 1.9
crazymoloch = 3.2
Qunnipiac = 3.8
Mason-Dixon = 4.9
Strategic Vision = 4.7
Public Policy Polling = 5.2
Rassmussen = 5.6
ARG = 12.0
Zogby = 12.0
Lower Plains/Upper South:
crazymoloch = 2.0
Zogby = 3.9
ARG = 4.8
Public Policy Polling = 6.1
Survey USA = 6.2
Mason-Dixon = 6.4
Rasmussen = 6.8
South (high African American Population):
Survey USA = 9.5
crazymoloch = 12.3
Strategic Vision = 13.5
Zogby = 14.5
Insider Advantage = 15.5
Public Policy Polling = 16.1
Rassmussen = 16.2
Southwest (Arizona only):
Rassmussen = 3.9
crazymoloch = 4.6
Mason-Dixon = 6.4
DETAILED RESULTS
.
COMMENTS:
(1) Looks like no one knows anything about polling when it comes to the Deep South. I’ve always wondered if we could expect astronomical black turnout with a black candidate at the top of the ticket. With Obama it now looks like this expectation could bear out. Georgia and Alabama might not be in play in Novermber, but unusually high African American turnout could make Republicans sweat at least a little in these states.
(2) Is SUSA good or what? A mean error of 1.9 points in each quantity across Democratic Strongholds! Damn! Not quite as unbelievably awesome elsewhere.
(3) Looks like everyone has a general idea of what they’re doing in the lower plain states, midwest and upper south. Since every pollster had respectable results in this region, my weighted poll averages were almost dead-on.
.
SPREADSHEET DATA