Sunday, October 29, 2006, the news broke: Jerry Brady, Democratic candidate for Governor of Idaho, was just one point behind his opponent. On Tuesday, we learned the Cook Political Report changed the race to Toss-Up status. Then Thursday, another poll, this time with the Democrat up by five points. In freakin' Idaho.
Then came election day, and the Republican won by about nine points.
In 2010, Daily Kos commissioned a poll of the governor's race in Hawai'i in early October. The result was a virtual tie, with the Democrat Neil Abercrombie leading by an insignificant two points. Rasmussen agreed. Two more polls showed margins that were only slightly better, and it looked like the Big Red Wave of 2010 might very well wash up on Hawai'i's shores.
Then came election day, and Abercrombie won by about seventeen points!
Michael Steele was just three points away from winning a Senate seat according to the polling average, only to lose by eleven points. Harry Reid pulled out a big come-from-behind victory in 2010. Pat Quinn was down by six in his quest to hold on to the Governor's seat in Illinois, but eked out a win. A string of Republican heartbreaks recently, where polls let them think they were close, but they were so very wrong.
How did all these polls - averages of multiple polls in each case - fail so spectacularly? Can we predict when it will happen again? And what does this have to do with Elizabeth Warren? Join me below...
So, about Elizabeth Warren...
Right now, Warren and Brown are tied in the polls. I expect the margin should shift around 5 points in her favor; and at a minimum, it should shift about 1 point, so if the election were held today, we would expect Warren to win.
Wait, WHAT? Where are these numbers coming from???
From a basic rule of thumb: for races for Senate and Governor, 95% of the time the tie goes to the Democrat in Blue States, and to the Republican in Red States. This rule does not apply to Swing States, Democratic Red States (LA, AR, KY, OK, WV), or contests with Third Parties >5%.
The corollary says beware small Republican leads in Blue States and small Democratic leads in Red States. They are likely (but not guaranteed) to evaporate. See below for details.
The nitty gritty.
This prediction is based on the comparison of polling averages and election results from 31 close elections (polling within 8 points) for Governor and Senate from 2004-2010. Races where third parties garnered more than 5% in the election were excluded (more about these later). Races where the state party or candidate did not align with the national party (for instance, West Virginia) were also excluded. Polling averages were simple averages of the last month of polls, or the last 10 days if a trend was evident. Research 2000 polls were excluded.
There's more error in polls for Governor and Senate.
Above I discussed a couple instances where the polls completely blew it. But it's not just the spectacular failures - on average, polls for the 31 close Senate and Governor races miss the margin by 3.2 points, while polls of 32 close Presidential races from 2004 and 2008 miss the margin by an average of 1.9 points.
Many people have noticed that polls for Governor and Senate have more error; in fact, it was mentioned at the Netroots Nation polling panel this year. But why?
Partisanship matters.
There's a simple explanation for most of the error in the Senate and Governor polls: the partisan tendency of the state. Here's what happens when you take those errors shown above and spread them out as a function of Obama's vote share in 2008:
That's a really nice correlation. Most of the error in these polls can be explained by partisan tendencies! This means we can predict when polls will fail. (Note that Hawaii and Idaho were excluded so as not to unduly influence the regression curve.) There's still plenty of error left over for things like last-minute campaign developments, voter turnout, too many undecideds, and other phenomena, of course.
Outsmarting the polls.
You can use this graph to see what the change in the margin between candidates will likely be between the polls and the election results. For instance, Obama got 57% of the vote in New Mexico. That means we would expect the margin of the 2012 Senate election to be about 3 points more Democratic than the polls, although anything from -2 to +8 wouldn't be surprising. Right now, the Democrat has a lead of 4 points; so this seat looks fairly safe for the time being.
Other contests to keep an eye on with regards to this effect this year are WA-Gov, HI-Sen, MI-Sen, NV-Sen, and WI-Sen.
Third party candidates throw a monkey wrench into the equation.
Now if we take the contests I excluded earlier, those where third parties received more than 5% of the vote, and plot them against the regression show above, we see things just aren't the same:
In many cases, it appears third parties attracted the support of conservative voters in the polls, who then either didn't vote or voted for the Republican after all. In Minnesota, the strong Independence party seems to change the dynamics completely, and a passel of local pollsters with consistently weird numbers doesn't help.
The most interesting election in this category is the 2010 Illinois election. Between the Senate and Governor's races, there were four unpopular major party candidates and five third party candidates of note. In the Senate race, the polls got it right; in the race for Governor, Democrat Pat Quinn came from six points down to win by a whisker. Here's PPP's final poll; if you can figure out what happened let me know!
But ya gotta know the territory!
Also excluded from the analysis were contests in Oklahoma, West Virginia, and Kentucky, where exit polls from 2004 and 2008 show about a third of Democrats voted for Bush or McCain. The local Democratic party is clearly not aligned with the national Democratic party in these states. Two contests were excluded because they included Republican candidates who are misaligned with the national party: the 2006 RI Senate contest (Chafee turned Independent), and the 2010 VT Governor contest (a borderline decision). Here's the graph where these five points fall. Interestingly, if you calculate the percent of West Virginian primary voters who vote in the Democratic primary, it's 65%. If you use this number instead of Obama's number, West Virginia's point falls right on the line.
__________
Beyond the Margin of Error is a series exploring problems in polling other than random error, which is what the margin of error measures.
Next up: How the 2008 Polling Was Often Wrong, and Will Be Again in 2012
