Skip to main content

Like many of the rest of you, I really enjoyed watching the right's meltdown this week.  While they were busy unskewing polls to avoid confronting reality, our side was busy winning the election.  But while this election was a broad triumph for science over ignorance, that doesn't mean that we shouldn't look at polling results critically.

The polls right before the election showed amazing agreement.  The last 30 polls released before the election all fit into a narrow 5% range from Obama +4 to Romney +1.  In fact, the only poll to show Obama +4 was a Democracy Corps poll, a Democratic pollster.  The only polls to show Romney leading were Gallup and Rasmussen, two pollsters with some of the strongest pro-Romney house effects. If we take these polls out, all of the polls fit into an astoundingly small 3% window.

This is fantastic considering some of the obstacles pollsters have to deal with:

Non-response bias:  Up to 9 out of 10 people refuse to answer polls
Sampling bias:  Are some types of voters more likely to answer polls?  Should we include cellphones or not?
Likely voter screen:  The mother of all polling dilemmas, how do we determine if someone will actually vote?

Yet, with all of these challenges, nearly all of our pollsters projected results within a tiny 3% window.  These results seem to be too good to be true.

In fact, the results may be too good to be true.  I'll let you decide.

We tend to judge polls unfairly.  If the last poll that a polling organization conducts before the election is close to the actual result, we consider them a good pollster.  If they miss by more than a few points, we call them bad.

But the truth is, if we are looking at only one poll, the results can vary wildly and it's not the pollster's fault.  Consider a poll that shows a 50-50 Obama-Romney tie with a 3% margin of error.  This means that 95% of the time, the result will be between Obama 53% Romney 47% (Obama +6) and Romney 53% Obama 47% (Romney +6) for a whopping 12 point swing in the winning margin.  And that's only if the pollster is lucky enough not to miss outside the margin of error.

Polls must miss outside the margin of error about 1 out of every 20 times.  This is not to say that if a pollster conducts 20 polls that exactly one poll will miss.  A pollster has about a 64% chance of missing at least once in 20 polls.  However, if we conduct 1000 polls,  we should miss on average about 50 of those polls.  The probability of missing on at least 40 polls is about 94%.  The probability of missing on at least 30 polls is about 99.85%

Let's limit our discussion to polls where all of the interviews took place in November.  October 31st was the day that president Obama surveyed the storm damage with governor Christie.  If there was a storm bounce, these polls should capture all of it.  There were 18 such polls:

Data Source:

For all of our polls, we're going to ignore the small number of undecided and "other" likely voters and focus on the margin of victory between Romney and Obama.

Of these 18 polls, not a single one missed the correct result by more than its margin of error.  This is a very good thing.  Most of our polls have about a 3% margin of error.  They're still counting votes, but right now, the election results are: Obama 50.6, Romney 47.9 (Obama +2.7%).  A poll with a perfect likely voter model just missing the margin of error in Romney's favor would show Romney +4.  On the other side, a poll missing in Obama's favor would show Obama +9.  Can you imagine if a pollster published these results the day before the election?  They would be laughed out of Pollster Town.  With 18 polls, assuming all of them had perfect models, there was a 60% chance that one of them would miss.  It's slightly lucky that nobody missed, but nothing overly unusual.

Let's go a step further.  A poll will miss by less than half of its margin of error about 32% of the time.  For a poll with a 3% margin of error, this means we can move up to 1.5% between the candidates.  Our narrower mini margin of error allows a result between Obama 49.1, Romney 49.4 (Romney +0.3%) and Obama 52.1, Romney 46.4(Obama +5.7%).   Of the 18 polls in our list, only two:  Rasmussen and Gallup missed by more than half of their margin of error.  A collective result this good or better only happens about 4% of the time.  

Let me repeat that:  assuming every single one of our pollsters has a perfect likely voter screen and has overcome a crippling non-response bias, results with this little statistical noise should only occur about 4% of the time.

But the models were not perfect.  The mean margin in our 18 polls is Obama +1%, an average error of 1.7%.  From our list of polls, not a single poll missed this average by more than half of its margin of error.  This is likely to occur only about 0.1% of the time.  Marvel at the narrow polling gap while you can, folks.  Results like this should only occur once every 4000 years in presidential elections!

Let's now go back in time.  There are 86 presidential polls listed on Real Clear Politics dating back to the beginning of August.  If we compare the winning margin in each of these polls to the RCP average the day before the poll was released, not one poll missed the RCP average by more than its margin of error.  For 86 polls we should expect about four misses but we had zero.  There is just over a 1% chance of having no polling misses.  How fortunate for our pollsters!

We put a lot of stock in our public polls.  They drive enthusiasm and donations in what has become a $6 billion election industry.  While we may treat one poll with cynicism, we tend to accept the wisdom of polling averages without question.  Maybe that should change.

There is a terrible lack of transparency in public polls.  Most polls don't release their RV results or their actual raw data.  We have no idea what goes into the secret sauce of a pollster's likely voter screen.  If a pollster wanted to put their thumb on the scale, they could produce just about any result they wanted.  How do we know that they didn't do it here?  How do we know that pollsters are releasing all of their polls and not holding back results that might appear embarrassing?  Are pollsters cheating and looking at other results before releasing their own?

I'm not saying that all pollsters are bad.  Some pollsters (PPP for example) take great pains to be transparent.  We need to be a little bit more wary of polling averages.  It's time to start asking polling companies some hard questions.

Originally posted to Cosecant on Sat Nov 10, 2012 at 08:18 AM PST.

Also republished by Community Spotlight.

Your Email has been sent.
You must add at least one tag to this diary before publishing it.

Add keywords that describe this diary. Separate multiple keywords with commas.
Tagging tips - Search For Tags - Browse For Tags


More Tagging tips:

A tag is a way to search for this diary. If someone is searching for "Barack Obama," is this a diary they'd be trying to find?

Use a person's full name, without any title. Senator Obama may become President Obama, and Michelle Obama might run for office.

If your diary covers an election or elected official, use election tags, which are generally the state abbreviation followed by the office. CA-01 is the first district House seat. CA-Sen covers both senate races. NY-GOV covers the New York governor's race.

Tags do not compound: that is, "education reform" is a completely different tag from "education". A tag like "reform" alone is probably not meaningful.

Consider if one or more of these tags fits your diary: Civil Rights, Community, Congress, Culture, Economy, Education, Elections, Energy, Environment, Health Care, International, Labor, Law, Media, Meta, National Security, Science, Transportation, or White House. If your diary is specific to a state, consider adding the state (California, Texas, etc). Keep in mind, though, that there are many wonderful and important diaries that don't fit in any of these tags. Don't worry if yours doesn't.

You can add a private note to this diary when hotlisting it:
Are you sure you want to remove this diary from your hotlist?
Are you sure you want to remove your recommendation? You can only recommend a diary once, so you will not be able to re-recommend it afterwards.
Rescue this diary, and add a note:
Are you sure you want to remove this diary from Rescue?
Choose where to republish this diary. The diary will be added to the queue for that group. Publish it from the queue to make it appear.

You must be a member of a group to use this feature.

Add a quick update to your diary without changing the diary itself:
Are you sure you want to remove this diary?
(The diary will be removed from the site and returned to your drafts for further editing.)
(The diary will be removed.)
Are you sure you want to save these changes to the published diary?

Comment Preferences

  •  The following is not true: (1+ / 0-)
    Recommended by:
    Of these 18 polls, not a single one missed the correct result by more than its margin of error.
    The Rassmussen Poll missed its margin of error, which was +/- 3. (Surprise, surprise!)

    Gallup has a higher margin of error, so it has not missed its margin of error yet.

    With votes still being counted on the West Coast, Gallup very well also could miss its margin of error and many of the polls that showed the race as being tied also could have missed their margin of error.

    As it stands right now, ABC/WashPost and Pew, which both had Obama at +3, were the best.

    "A candle loses nothing by lighting another candle" - Mohammed Nabbous, R.I.P.

    by Lawrence on Sat Nov 10, 2012 at 09:02:28 AM PST

    •  you need to double the margin of error (2+ / 0-)
      Recommended by:
      Lawrence, raster44

      when talking about head-to-head numbers.  Ras shows Romney +1.  If we take 3 points away from Romney and give them to Obama, we have O+5, so Ras is still within the margin of error.  

    •  The MoE is relevant only ... (1+ / 0-)
      Recommended by:

      with regard to an unbiased sample. It's fairly clear that Ras uses a likely voter model that skews R (they appear to be off in a systematic way). So the Ras poll isn't a great counter-example.

      But the Diary raises a question that has been lurking in the back of my head for a while, too. Yes, if we take averages of polls the MoE should shrink because the effective sample size has increased (assuming, heroically but apparently effectively, that differences in sampling methodology and weighting procedures between pollsters cancel out). But the degree to which the pollsters all were on the "right" side of this contest, not just at the national level but also at the state level, is fairly extraordinary.

      Indeed, I'm astonished at how well the polls have done. Gobsmacked. Response rates for many polls were down to 2-3%; it's the rare poll, I think (but correct me if I'm wrong) that succeeds with a 9% response rate. That responders appear to be representative of non-responders is desperately surprising to me.

      •  Hey, they can get lucky sometimes (1+ / 0-)
        Recommended by:

        Can't they?

        Remember, that a polling firm worth its salt (PPP, for example) re-evaluates constantly, always trying to improve their model. The problem is that polling in this country is just so damn difficult. You can work very, very hard to get very few responses, many of whom may be exceptions that skew your results (i.e. landlines skewing R, cell phones skewing D).

        “Never attribute to malice that which can be adequately explained by stupidity.” - Hanlon's Razor

        by Mister Black on Sat Nov 10, 2012 at 03:24:47 PM PST

        [ Parent ]

    •  final numbers (4+ / 0-)

      I only have one issue here.  That is there are still i great deal of votes that have not been counted in CA, OR, and WA state.  I cant determine just how many but i could be 4 or 5 million.  So BO's margin of winning could easily go from 3 million and 2.7% to as much as 4.5 million and closer to 4%.

      In addition, if you look at the swing state polls BO won by much bigger margins than the polls showed in IA, WS, CO, NV, VA, and NH. and in some cases by several %.  For example BO won WS by 6%, IA by 6%, CO by 4%, NH by 6%, and VA by 3%.  These were much higher that the state averages on Huff post and the other polling composite sites.  In the case of CO, IA, and VA by quite a bit more.  

      Thus, i would make the argument that if fact almost all the polls except for Dem Corps were wrong and as a whole it was not Rommey who was being under polled it was BO and this was seen in both the national polls and ever more in the swing state polls.  Just my opinion.

  •  No comment to all pollsters (1+ / 0-)
    Recommended by:

    I had a few dozen polling robocalls and a handful of human polling calls.   Nearly every call for polling was either from an unknown source, CrossRoads or Americans for Prosperity.

    I did not answer a single poll question via phone and nearly all web polls with one exception.  OFA (Obama for America) robocalled with a few questions which I answered.

  •  Check my math (and logic) (3+ / 0-)
    Recommended by:
    raster44, KJG52, demimondian

    Each poll had a 95% chance of being accurate.
    18 polls.
    The odds of all of them being accurate should be .95^18 (.95 to the 18th power)
    .95^18 = about 40% .... so not much worse than a coin flip to get all 18 within MOE.

    What I found more interesting was that at least 15 of the 18 UNDER estimated BHO's margin.  I am getting past my comfort zone, but I think the probability of a 3/15 split out of 18 trials is pretty small.... So the "house effect" of all polls seems to lean R.

  •  Two things (0+ / 0-)

    (1) Denver11 is correct if the polls are independent of one another.  I seem to remember reading that there's some evidence that polls tend to move towards one another at the end of the cycle, which would make the probability of extreme performance (either good or bad) higher still.

    (2) An MOE estimate is just an estimate.  It's just a sample statistic, and, like any sample statistic, can be wrong.  That's not as important here as the previous point, but it's important.

Subscribe or Donate to support Daily Kos.

Click here for the mobile view of the site