Skip to main content

Sunday, October 29, 2006, the news broke: Jerry Brady, Democratic candidate for Governor of Idaho, was just one point behind his opponent.  On Tuesday, we learned the Cook Political Report changed the race to Toss-Up status.  Then Thursday, another poll, this time with the Democrat up by five points.  In freakin' Idaho.

Then came election day, and the Republican won by about nine points.

In 2010, Daily Kos commissioned a poll of the governor's race in Hawai'i in early October.  The result was a virtual tie, with the Democrat Neil Abercrombie leading by an insignificant two points.  Rasmussen agreed.  Two more polls showed margins that were only slightly better, and it looked like the Big Red Wave of 2010 might very well wash up on Hawai'i's shores.  

Then came election day, and Abercrombie won by about seventeen points!

Michael Steele was just three points away from winning a Senate seat according to the polling average, only to lose by eleven points.  Harry Reid pulled out a big come-from-behind victory in 2010.  Pat Quinn was down by six in his quest to hold on to the Governor's seat in Illinois, but eked out a win.  A string of Republican heartbreaks recently, where polls let them think they were close, but they were so very wrong.  

How did all these polls - averages of multiple polls in each case - fail so spectacularly?  Can we predict when it will happen again?  And what does this have to do with Elizabeth Warren?  Join me below...

So, about Elizabeth Warren...
Right now, Warren and Brown are tied in the polls.  I expect the margin should shift around 5 points in her favor; and at a minimum, it should shift about 1 point, so if the election were held today, we would expect Warren to win.  

Wait, WHAT?  Where are these numbers coming from???
From a basic rule of thumb: for races for Senate and Governor, 95% of the time the tie goes to the Democrat in Blue States, and to the Republican in Red States.  This rule does not apply to Swing States, Democratic Red States (LA, AR, KY, OK, WV), or contests with Third Parties >5%.

The corollary says beware small Republican leads in Blue States and small Democratic leads in Red States.  They are likely (but not guaranteed) to evaporate.  See below for details.

The nitty gritty.
This prediction is based on the comparison of polling averages and election results from 31 close elections (polling within 8 points) for Governor and Senate from 2004-2010.  Races where third parties garnered more than 5% in the election were excluded (more about these later).  Races where the state party or candidate did not align with the national party (for instance, West Virginia) were also excluded.  Polling averages were simple averages of the last month of polls, or the last 10 days if a trend was evident.  Research 2000 polls were excluded.

There's more error in polls for Governor and Senate.
Above I discussed a couple instances where the polls completely blew it.  But it's not just the spectacular failures - on average, polls for the 31 close Senate and Governor races miss the margin by 3.2 points, while polls of 32 close Presidential races from 2004 and 2008 miss the margin by an average of 1.9 points.  

Many people have noticed that polls for Governor and Senate have more error; in fact, it was mentioned at the Netroots Nation polling panel this year.  But why?

Partisanship matters.
There's a simple explanation for most of the error in the Senate and Governor polls: the partisan tendency of the state.  Here's what happens when you take those errors shown above and spread them out as a function of Obama's vote share in 2008:

That's a really nice correlation.  Most of the error in these polls can be explained by partisan tendencies!  This means we can predict when polls will fail.  (Note that Hawaii and Idaho were excluded so as not to unduly influence the regression curve.) There's still plenty of error left over for things like last-minute campaign developments, voter turnout, too many undecideds, and other phenomena, of course.  

Outsmarting the polls.
You can use this graph to see what the change in the margin between candidates will likely be between the polls and the election results.  For instance, Obama got 57% of the vote in New Mexico.  That means we would expect the margin of the 2012 Senate election to be about 3 points more Democratic than the polls, although anything from -2 to +8 wouldn't be surprising.  Right now, the Democrat has a lead of 4 points; so this seat looks fairly safe for the time being.

Other contests to keep an eye on with regards to this effect this year are WA-Gov, HI-Sen, MI-Sen, NV-Sen, and WI-Sen.

Third party candidates throw a monkey wrench into the equation.
Now if we take the contests I excluded earlier, those where third parties received more than 5% of the vote, and plot them against the regression show above, we see things just aren't the same:

In many cases, it appears third parties attracted the support of conservative voters in the polls, who then either didn't vote or voted for the Republican after all.  In Minnesota, the strong Independence party seems to change the dynamics completely, and a passel of local pollsters with consistently weird numbers doesn't help.  

The most interesting election in this category is the 2010 Illinois election.  Between the Senate and Governor's races, there were four unpopular major party candidates and five third party candidates of note.  In the Senate race, the polls got it right; in the race for Governor, Democrat Pat Quinn came from six points down to win by a whisker.  Here's PPP's final poll; if you can figure out what happened let me know!

But ya gotta know the territory!
Also excluded from the analysis were contests in Oklahoma, West Virginia, and Kentucky, where exit polls from 2004 and 2008 show about a third of Democrats voted for Bush or McCain.  The local Democratic party is clearly not aligned with the national Democratic party in these states.  Two contests were excluded because they included Republican candidates who are misaligned with the national party: the 2006 RI Senate contest (Chafee turned Independent), and the 2010 VT Governor contest (a borderline decision).  Here's the graph where these five points fall.  Interestingly, if you calculate the percent of West Virginian primary voters who vote in the Democratic primary, it's 65%.  If you use this number instead of Obama's number, West Virginia's point falls right on the line.

__________
Beyond the Margin of Error is a series exploring problems in polling other than random error, which is what the margin of error measures.

Next up: How the 2008 Polling Was Often Wrong, and Will Be Again in 2012

Originally posted to dreaminonempty on Fri Jun 15, 2012 at 06:17 AM PDT.

Also republished by Community Spotlight.

EMAIL TO A FRIEND X
Your Email has been sent.
You must add at least one tag to this diary before publishing it.

Add keywords that describe this diary. Separate multiple keywords with commas.
Tagging tips - Search For Tags - Browse For Tags

?

More Tagging tips:

A tag is a way to search for this diary. If someone is searching for "Barack Obama," is this a diary they'd be trying to find?

Use a person's full name, without any title. Senator Obama may become President Obama, and Michelle Obama might run for office.

If your diary covers an election or elected official, use election tags, which are generally the state abbreviation followed by the office. CA-01 is the first district House seat. CA-Sen covers both senate races. NY-GOV covers the New York governor's race.

Tags do not compound: that is, "education reform" is a completely different tag from "education". A tag like "reform" alone is probably not meaningful.

Consider if one or more of these tags fits your diary: Civil Rights, Community, Congress, Culture, Economy, Education, Elections, Energy, Environment, Health Care, International, Labor, Law, Media, Meta, National Security, Science, Transportation, or White House. If your diary is specific to a state, consider adding the state (California, Texas, etc). Keep in mind, though, that there are many wonderful and important diaries that don't fit in any of these tags. Don't worry if yours doesn't.

You can add a private note to this diary when hotlisting it:
Are you sure you want to remove this diary from your hotlist?
Are you sure you want to remove your recommendation? You can only recommend a diary once, so you will not be able to re-recommend it afterwards.
Rescue this diary, and add a note:
Are you sure you want to remove this diary from Rescue?
Choose where to republish this diary. The diary will be added to the queue for that group. Publish it from the queue to make it appear.

You must be a member of a group to use this feature.

Add a quick update to your diary without changing the diary itself:
Are you sure you want to remove this diary?
(The diary will be removed from the site and returned to your drafts for further editing.)
(The diary will be removed.)
Are you sure you want to save these changes to the published diary?

Comment Preferences

  •  Interesting (6+ / 0-)

    Maybe this will put to rest the idea (most recently in Wisc) that variances between polls and votes are evidence of Republicans stealing elections.

    •  Wisconsin polling average was right on target (15+ / 0-)

      The polling average was aroung R+6, actual result R+7.  I still excluded it because I started the analysis before the election was complete!

      •  Do these apply only for... (3+ / 0-)
        Recommended by:
        MichaelNY, caul, Wary

        ...Presidential cycles?

        I can understand that there may be Obama voters who stayed home in WI. But I don't think anyone is going to go to the polls in MA, vote for Obama, then vote for Scott Brown.

        •  Anyone? (3+ / 0-)
          Recommended by:
          caul, HudsonValleyMark, James Allen

          Sure, some people will. I guarantee it. In fact, I fully expect Brown to lose by a much smaller margin than Romney. Don't you?

          Formerly Pan on Swing State Project

          by MichaelNY on Fri Jun 15, 2012 at 10:17:14 PM PDT

          [ Parent ]

        •  All cycles (1+ / 0-)
          Recommended by:
          MichaelNY

          There is some weak evidence that the effect is more pronounced for presidential cycles, but there are too few data to tell for sure.

          •  Some data from 2000 (1+ / 0-)
            Recommended by:
            MichaelNY

            Photobucket

            Have to be careful with 2000 data because there is very real evidence of late movement - though you can argue that had to do with Nader.

            I am sure everyone is wondering, so the average Florida polling in the last week showed Bush ahead by .6667.

            I have always found it fascinating that the September polling in 2000 is actually more accurate in close states than the October or the late polling.

            You can certainly see some support for your argument.  It is worth remember Karl Rove actually misjudged California and New Jersey because of this.

            Send me a message - I want to share data (I have some '96 and '92 data, but not the 2010 data).

            I should also add there is a HUGE problem developing with cell phones.  I exchanged a tweet with Charles Franklin - who said polling right now is a black swan event waiting to happen.

            The bitter truth of deep inequality has been disguised by an era of cheap imported goods and the anyone-can-make-it celebrity myth - Polly Toynbee

            by fladem on Sat Jun 16, 2012 at 09:13:33 AM PDT

            [ Parent ]

            •  You're getting ahead of me! (0+ / 0-)

              That's the plotline of the next diary!  But it's interesting to see it holds up for 2000; I only went back to 2004.

              You may also remember Rove misjudged Hawai'i and sent Dick Cheney flying out there...!

              As far as data, I'm afraid I have to disappoint you.  I don't have a nice, fat spreadsheet of 2010 polling, just  averages of a handful of close races.

              There is definitely a problem with cell phones.  But I have noticed pollsters solving this problem by... calling cell phones anyway.  I've gotten automated polling calls on my cell phone more than once.

        •  Oh HELL yes they will (2+ / 0-)
          Recommended by:
          MichaelNY, Wary

          Take it from a MA resident: Brown's whole "independent" schtick works in this state, for a significant number of people. Obama will get a far larger percentage of the vote than Warren will, guaranteed.

          Obama will probably break 60% in MA. Warren won't come close to that. We have such a reputation as a "blue" state, people forget that our largest voting bloc is actually "unenrolled". And those unenrolled folks love splitting tickets--it makes them feel all important and independenty and shit. It's how we ended up with 12 years of GOP governors in this "blue" state.

          "Maybe: it's a vicious little word that could slay me"--Sara Bareilles

          by ChurchofBruce on Sat Jun 16, 2012 at 11:36:25 AM PDT

          [ Parent ]

    •  Exit polls are a different story (8+ / 0-)

      Exit polls are usually pretty accurate, mostly because they are polling people who actually voted and not trying to determine who is a "likely voter".  I assume you are referring to the exit polls in Ohio and Florida in 2004 which showed Kerry winning.  There are a number of studies that suggest that the odds of two exit polls being off as much as they seemed to be are pretty slim.  This does suggest something was a foot, but that is ancient history.  

      The difficulty lies not so much in developing new ideas as in escaping from old ones! - John Maynard Keynes

      by Do Something on Fri Jun 15, 2012 at 03:37:44 PM PDT

      [ Parent ]

      •  a bunch of exit polls in 2004 were way off (5+ / 0-)
        Recommended by:
        Larsstephens, MichaelNY, fladem, caul, Wary

        For instance, the New York exit poll showed Kerry winning by 31 points, which made no sense -- New York was blue, but not that blue. There's no question that the polls were off by more than random sampling error could explain.

        (The Florida exit poll actually showed Bush very slightly ahead, but it was still well off given the sample size, as you say.)

        •  Everything depends (4+ / 0-)

          On who you ask.

          A truly random sample can give you totally screwy and very far off results.

          Why? Because random isn't necessarily representative.

          Pollsters go to a lot of effort to make thier samples representative of the electorate as a whole, NOT just random.

          You could take a random sample and get 60% republican, in an area where the electorate is pretty even. Or one that is mostly over 60, when the electorate might be only 35% over 60.

          Get a sample like that, and the results will be way, way off.

          •  to try to be more specific... (2+ / 0-)
            Recommended by:
            Larsstephens, caul

            If it were possible to draw a simple random sample of voters, it's true that the sample could be "very far off," but the normal curve would govern how far off it is likely to be. Actually exit polls use a sample of precincts, and then samples of voters within those precincts, so the problem is more complicated -- but still, if random error were the only issue, it would be easier to characterize how large it might be.

            Exit pollsters work pretty hard to choose precincts that, together, will be representative of the electorate. Then they work to get random samples within those precincts (except in precincts where they can try to survey everyone), and to compensate for obvious biases. For instance, in 2004, older voters were markedly less willing to complete the surveys.

            I'm not sure what you have in mind here:

            You could take a random sample and get 60% republican, in an area where the electorate is pretty even.
            Any random sample can be just plain off -- and it's a hard problem to deal with, because the pollsters can't assume that they know the party composition of people who voted. (Even if turnout by party registration is available, registration doesn't necessarily correspond with self-identified party ID.)

            But the error in 2004 couldn't (plausibly) have been just random (nor can most of it plausibly be attributed to vote miscounts and such). Clearly some other things were going wrong.

            If the results from a precinct are facially bizarre, for whatever reason(s), then the pollsters can downweight or ignore them.

    •  What planet do you spend most of your time on? n/t (0+ / 0-)

      Busting the Dog Whistle code.

      by Mokurai on Sat Jun 16, 2012 at 10:30:09 AM PDT

      [ Parent ]

    •  Until we have a paper trail for elections... (1+ / 0-)
      Recommended by:
      MichaelNY

      they will always have issues.  The head of the conservative movement states it here... plus track FL and recent issues ongoing...

      “I don’t want everybody to vote. Elections are not won by a majority of people, they never have been from the beginning of our country and they are not now.

      As a matter of fact, our leverage in the elections quite candidly goes up as the voting populace goes down.”

      -- Paul Weyrich, “founding father of the modern conservative movement,”1980

             The GOP says YOU have to have an ID to vote...
                                             but
              ....Million dollar donors should remain (cough) anonymous.
  •  I hope this is still the case (17+ / 0-)

    On the one hand, I remember another instance of this - Mittens vs Kennedy for MA Senate back in 1994.  Kennedy was reeling somewhat from the William Kennedy Smith scandal, it was a big Republican wave year, and there was a backlash against the Boston Globe for doing a hatchet job on Romney and Mormonism.  Kennedy slightly recovered by losing weight, quit drinking and married Victoria Reggie, but was up only 4 points in the polls.  And won by 14, 57-43.

    On the other hand, I don't think we saw this at all 2 years ago when Coakley lost to Brown. She was down in the polls and did indeed lose, I don't believe she did any better than the polls suggested.  

    I'd like to see Warren up and polling a good 4-5 points higher than Brown by election time, I don't want to have to count on this "blue state" bounce.

    Liberalism is trust of the people tempered by prudence. Conservatism is distrust of the people tempered by fear. ~William E. Gladstone, 1866

    by absdoggy on Fri Jun 15, 2012 at 06:35:39 AM PDT

    •  Special elections are ignored by (5+ / 0-)
      Recommended by:
      NYFM, Nulwee, Larsstephens, MichaelNY, caul

      a large Dem demographic.

      Romney - 2012 - He's A Trooper!

      by kitebro on Fri Jun 15, 2012 at 06:48:11 AM PDT

      [ Parent ]

    •  Still worked in Coakley versus Brown (6+ / 0-)
      Recommended by:
      NYFM, Matt Z, Nulwee, Larsstephens, MichaelNY, caul

      Coakley was down by about 9 points in the average of the final week of polling, and lost by only 5.   There were two problematic polls that got people's hopes up, showing a tie and Coakley up by 8.  Both of those were done by the discredited Research 2000.  All other polls had Brown leading.

      This effect appears to be still in working order for 2012, based on some analysis I'll show in future diaries in this series.  

      Counting on a blue state bounce - well, I think we can all agree that having Warren up by 4-5 points will let us sleep easier than polls showing a tie!

    •  And, is MA really a blue state? (7+ / 0-)

      Do they still warrant being called a blue state, after electing Romney and Brown?  Nice to see that Republicans are an afterthought in the state legislature, but statewide races no longer seem so simple.

      I am become Man, the destroyer of worlds

      by tle on Fri Jun 15, 2012 at 07:40:18 AM PDT

      [ Parent ]

      •  Yes. Absolutely. (11+ / 0-)

        Despite Romney, Brown and Brooke. For anyone who has ever lived there its blueness is very well known.

        But there remains a very strong streak of contrarian behavior. I swear that Republicans sometimes get elected just because everyone else in the country thinks they should not. Coakely was "supposed" to be elected, so the voters gave the rest of the country the finger and elected Brown. Just because.

        •  Yes it does, and Jeebus on Ed Brooke (6+ / 0-)
          Recommended by:
          drjat42, ybruti, Larsstephens, MichaelNY, caul, tle

          Talk about those that would not be Repubs today.  Ed Brooke was pro-choice, authored the Fair Housing Act, fought against Nixon to keep the Job Corps, fought against Nixon on weakening the EEOC and led the efforts to block 2 right-wing Supreme Court nominees.
          In 1975, he led efforts to keep the Voting Rights Act intact, as well as Title IX.

          I would probably also put William Weld in this category.

          I don't know, the whole Weld - Celluci/Swift - Romney thing.  I would mostly characterize it as coming from infighting among Democrats.  Belotti was the convention winner in 1990 against Weld, but then some of the other candidates threw their support to John Silber (who was last at the convention) out of spite, so Silber won the primary.  He was an arrogant asshole who turned off everyone.  

          Same thing happened with Celluci - Atty General Harshbarger won the primary for the Dems, but the other 2 candidates McGovern and Donnelly had about 1/2 the vote combined, and they did nothing to support Harshbarger.

          Same thing again with Mittens - 5 way primary for Dem, including Robert Reich (who, having pissed off Clinton with his book, didn't even get Clinton's support).  Treasurer Shannon O'Brien won, but again had no support from the other candidates' backers, and took a hit from the fact that the state lost big $$ in the tech stock meltdown in 2000

          Dems in MA need to stop the squabbling and coalesce around the candidate.

          Liberalism is trust of the people tempered by prudence. Conservatism is distrust of the people tempered by fear. ~William E. Gladstone, 1866

          by absdoggy on Fri Jun 15, 2012 at 08:51:39 AM PDT

          [ Parent ]

          •  Thanks. (1+ / 0-)
            Recommended by:
            MichaelNY

            That's historical detail I wouldn't have a clue about.  Your conclusion seems reasonable,

            Dems in MA need to stop the squabbling and coalesce around the candidate.
            but, would that go against their DNA?  :-)

            I am become Man, the destroyer of worlds

            by tle on Sat Jun 16, 2012 at 06:50:23 AM PDT

            [ Parent ]

        •  It's not contrarian. (3+ / 0-)
          Recommended by:
          MichaelNY, caul, tle

          The MA state Democratic party is viewed as corrupted. Not actually on the take, but beholden to entrenched interests. Machine politics. The Bulger Brothers.

          That is how Coakley got nominated in the first place.

          But when presented with a Better Democrat (like Warren) I think we can rely on MA to do the right thing.

          •  I hope so. (1+ / 0-)
            Recommended by:
            MichaelNY

            Your comment is the most encouraging one I've read yet.  With such overwhelming dominance, it's hard for a group to not be corrupt.  In this case, I think it's the reverse;  Brown, as the hatchet man for the banks, is, IMO, deeply corrupt, while Warren is as close to squeaky clean as anyone is likely to get.  Which is no doubt why the puppet masters hate her so much.

            I just wish she would stop being so scripted in interviews.  

            I am become Man, the destroyer of worlds

            by tle on Sat Jun 16, 2012 at 06:42:06 AM PDT

            [ Parent ]

      •  Oregon is a pretty blue state but it does elect (6+ / 0-)

        occassional Republicans as does California and Washington. Most of the time it is because the Dem running is unlikable,simply incomprehensible or too radical. There are people who are near the center and waver based on how disturbing or stimulating the pol is. Even red states find some righties too right or radical and will vote for a Dem who isn't too far away from thier thoughts.

        A lot of people are looking for stability in government and what they can expect from pols. If you live near the edge, sudden radical reform can translate into a disaster to you.

        I support Warren and think she is going to win.

        Proud Slut...Fear is the Mind Killer

        by boophus on Fri Jun 15, 2012 at 08:25:47 AM PDT

        [ Parent ]

  •  I don't think polls will accurately predict these. (5+ / 0-)

    close races until pollsters start calling cell phone users. The majority of voters under 51 years of age don't use land lines anymore.

    I know it costs more but pollsters are getting plenty of work these days. They can afford it.

    Thanks for the astute analysis.

    "That men do not learn very much from the lessons of history is the most important of all the lessons of history." ~ Aldous Huxley

    by markthshark on Fri Jun 15, 2012 at 06:37:50 AM PDT

    •  I don't think cell phones are part of this effect (8+ / 0-)

      The reason is there's no change in this effect from 2004 to today.  Any cell-phone related effect should have changed dramatically since 2012.

      That being said, there's certainly a whole basketful of other errors related to cell phones and polling.

      •  I think the problems are hidden by voter apathy (2+ / 0-)
        Recommended by:
        Larsstephens, caul

        Cell phone users, who happen to be young, are also less likely to turn out to vote. Even if the pollsters undersample cell phones, they make up for that by young people turning out less than expected most elections.

        That said, i bet the better pollsters already try to correct for both of these issues when they extrapolate from the raw data.

        The bigger problem to me would seem to be the hang-up effect and party i.d. We know party i.d. fluctuates with how many people are excited about a party/candidate, and we know that the likelihood of answering a pollster are also a function of the excitement about voting. Then how do we get an accurate handle on the party i.d. of the country used to correct for the sample bias? Again, apathy tends to correlate with voting behavior, so they are lucking out, but when it's interpreted, I'm not sure the interpretations are really getting at the underlying issues going on (and lead to overemphasis of the "centrists" or "independents", when most of those folks are actually already partisan). Late-breaking events that swing elections imo are more about getting apathetic voters to turn out for their side, rather than swaying a voters' opinion about something. But how to tease those apart given the also known issues with exit polling answering rates?  

        Maybe these things so rarely diverge, it's not worth it to worry about the real issues going on.

        •  one other point (2+ / 0-)
          Recommended by:
          Larsstephens, caul

          The reason these things do matter aside from polling interpretation accuracy is that the interpretation can effect what parties decide to do to win elections. I think Democrats have either chosen very poorly to focus on the issues center in many years, instead of the apathetic partisan, or else they long ago determined these folks are too difficult to reach. Republicans have long ago decided to focus on base turnout, particularly in the more apathetic non-presidential years, and don't worry about converting "centrists" (that don't really exist in real numbers imo). They've sometimes swayed too far, and turned off their own partisans, but I don't think they swayed many people to the other side until they really get dangerous.

          So their ads are not about swaying someone's opinion (which we often grouse about on here, including me, "why would this change anyone's mind") and more about getting their apathetic supporters to get mad and come out and vote against the other guy.

      •  It's not the cell phones (3+ / 0-)
        Recommended by:
        Larsstephens, MichaelNY, caul

        It's that and everything else.

        You have to get a representative sample. In a single state, or a single district, you might call every landline in the district and still not get a representative sample. Even calling cell phones might not give you enough.

        Response rates on phone surveys have dropped from 70% to under 10. When only one in ten even responds to,the survey, it can be almost impossible to get a statistically valid sample.

    •  Many of the big pollsters do (3+ / 0-)
      Recommended by:
      HudsonValleyMark, MichaelNY, caul

      call cell phones.

      You are reading my signature line. #hashtag

      by cardinal on Fri Jun 15, 2012 at 04:58:36 PM PDT

      [ Parent ]

  •  This lines up with my thinking (12+ / 0-)

    In a Presidential year, in Massachusetts, Brown is doomed.

    Brown won the special election due to a combination of factors (special election, weak Democratic candidate, Tea Party), most of which won't be present in 2012.  It's Massachusetts and I just can't see them electing a Republican to a full Senate term.

    28, white male, TX-26 (current), TN-09 (born), TN-08 (where parents live now)

    by TDDVandy on Fri Jun 15, 2012 at 06:40:44 AM PDT

  •  Last weekend 5-10% deciders (3+ / 0-)

    One of the problems of polls, is that there usually is a group of 5-10% of the electorate that doesn't make up it's mind until the last weekend or even as they walk into the voting booth.  To me, if you haven't made up your mind before the final weekend, you haven't been paying attention and you're making your decision on a gut feeling, probably based on Media Ads.  This extremely irrational behavior can throw the final results of any poll.

  •  Did you happen to find a difference (2+ / 0-)
    Recommended by:
    Larsstephens, MichaelNY

    Between presidential elections and midterms? I would guess the impact might be exacerbated by the top of the ticket in 2004 and 2008.

    "What do you mean "conspiracy"? Does that mean it's someone's imaginings and that the actual polls hovered right around the result?" - petral

    by conspiracy on Fri Jun 15, 2012 at 07:23:34 AM PDT

  •  Very cool and extra geeky (3+ / 0-)
    Recommended by:
    Larsstephens, MichaelNY, bythesea

    and NO that is NOT an oxymoron

  •  Another reason for more error (7+ / 0-)

    in Senate/Gov races might be less knowledge.

    It's hard not to have heard of Obama and Romney.

    But I bet a lot of people haven't heard of their senator or governor, and even more haven't heard of the challenger.  But who wants to say that to a pollster? Some polls don't even let you say it.

  •  Math! (10+ / 0-)

    I have a dumbed-down version of this. It's called the "how will they get to 50%?" rule. Heidi Heitcamp is looking good at 46-ish. But in North Dakota, how will she get to 50?

    Best example is Joe Donnelly in Indiana. He's tied in the low 40s with his Republican opponent in all the latest polling. Awesome! I guess. It's Indiana. It's much easier for Team Red to get over 50% than our guy.

    Scott Brown is in the same place, stuck in the low- to mid-40s. How will he get to 50 in Mass?

    It's nice to see math backing up my simplistic formulation.

    •  so, undecideds break to the prevailing party (4+ / 0-)
      Recommended by:
      Larsstephens, MichaelNY, pademocrat, caul

      That makes a lot of sense. Granted, "undecideds" is a pretty murky concept -- different pollsters get very different results. Also, polling taps some people who are just guessing who they support, but will figure it out when they see the party IDs on the ballot.

      I bet the analysis here could pretty easily be extended to see whether the proportion of undecideds predicts the extent of the "break." Ideally one would control for house effects: I imagine that some pollsters consistently get more undecideds (at least in down-ticket contests) than others.

      •  Not necessarily (7+ / 0-)

        Undecideds may not vote at all, and Democrats who say they'll vote for the Republican may just change their minds when they see the ballot.  I looked at a relationship between undecideds and the magnitude of this effect and couldn't find one.  

        There'll be more on this topic in the next diary in the series.  :)

        •  I guess you already answered my question (4+ / 0-)
          Recommended by:
          Larsstephens, MichaelNY, pademocrat, caul

          The house effects could complicate looking for this effect -- but if it isn't there, it isn't there.

          ...Democrats who say they'll vote for the Republican may just change their minds when they see the ballot.
          It was presumptuous for me to attribute that to guessing versus knowing the party IDs, but I suspect that is a big reason. Surely not the only one, especially because it doesn't apply when party ID is supplied in the poll. I bet ballot design has a significant impact on people's propensity to vote straight-party tickets.
          •  I took a quick look at ballot order (3+ / 0-)
            Recommended by:
            HudsonValleyMark, Larsstephens, caul

            Trying to sort out the Illinois election, I thought maybe ballot order  might have had an effect.  It looked like being first on the ballot might have given that candidate an extra 1%, but then a couple races weren't consistent with that idea, and I gave up.  I'd guess somebody has studied the effect of ballot design, somewhere, and in fact I think I've read something about it, but I have no idea where.

            •  the effect shd depend on other factors (0+ / 0-)

              I haven't focused on these questions at all, which is why I'm in brainstorm mode instead of lit-citation mode. But intuitively, appearing first will matter more if people don't have much other basis for deciding (yet decide to vote in the contest anyway). That could depend in part on the candidates, the campaign, and the ballot design (e.g. the prominence of party ID). I have no idea how that might apply in the Illinois election -- not even sure which one you mean!

              •  Sorry.. (1+ / 0-)
                Recommended by:
                HudsonValleyMark

                The 2010 Illinois election I mentioned in the text.  Nobody liked any of the major party candidates for Governor or Senate (well, not nobody, but they both had lousy favorability numbers), which made for a real wacky election.  A fair number of polls with 10-20% undecided, and/or 10% third party candidates.

                •  oh, of course (0+ / 0-)

                  So, you might be able to come up with a story about why ballot order mattered more in one contest than the others -- and it might be true, or somewhat true, although it would be hard to assess it vis-a-vis other stories.

                  If different Illinois counties use different ballot designs (even with the same orders), conceivably that could lead to interesting results, although it would be hard to deal with confounding factors. Of course this has nothing to do with your point in the diary!

            •  I have to imagine that (1+ / 0-)
              Recommended by:
              MichaelNY

              ballot placement does matter, but it seems like it would have a greater effect in primaries than general elections.

              Male, currently staying in Osaka-01. Voting in RI-01, went to college in IL-01.

              by sapelcovits on Sat Jun 16, 2012 at 04:57:07 AM PDT

              [ Parent ]

              •  4-5% more likely to win in local elections (1+ / 0-)
                Recommended by:
                MichaelNY

                Here's a study of thousands of local California elections claiming whoever's listed first is 4-5% more likely to win because of it (in low information local elections)

                And here's a summary of studies showing ballot order effect, generally indicating little effect for 'Big Ticket' elections but 1-2% extra for the first listed for primaries, House seats, city council, etc.

                •  yeah, it would definitely have a bigger (1+ / 0-)
                  Recommended by:
                  MichaelNY

                  effect on low info races where people literally pick the first name they see. That's my theory for how a slacker like Adam Gussen could win the NJ-05 (D) primary earlier this month.

                  Male, currently staying in Osaka-01. Voting in RI-01, went to college in IL-01.

                  by sapelcovits on Sat Jun 16, 2012 at 07:00:29 AM PDT

                  [ Parent ]

    •  That's a pretty good way to put it. (2+ / 0-)
      Recommended by:
      MichaelNY, bythesea

      The "How will they get to 50?" rule should work as long as the polls don't have a huge amount of undecideds.

      I wrote this up as a way to quantify the 'feel' that manic poll watchers, such as those who hang out here, tend to develop, and to check to see if it was correct.  Now we know for sure you all are right.  :)

      •  Here's an example (2+ / 0-)
        Recommended by:
        MichaelNY, bythesea

        ...where undecideds could screw up the "How will they get to 50?" rule:  A poll just before the 2010 Oregon Governor's election showed Kitzhaber (D) at 43.  If you just look at how Kitzhaber would get to 50, it seems like a bit of an uphill slog, but there were 10% undecided.  Looking at the poll's D-R margin of -3, the graph above says on average the polling margin is off by about 3 points in a state as blue as Oregon, so you can tell there's a decent chance of Kitzhaber winning, which he did.  (Thanks to James Allen whose comment above led me to this example.)

    •  exactly. (2+ / 0-)
      Recommended by:
      MichaelNY, bythesea

      And particularly in Massachusetts, Brown was originally elected in a special election in January, and not by a wide margin.  Is it actually possible in today's Massachusetts, given the turnout in a presidential election, for a Republican to win a statewide election in a presidential election while the Republican nominee is losing by 20+ points?  

      Lewis & Clark Law class of 2015

      by James Allen on Fri Jun 15, 2012 at 04:33:57 PM PDT

      [ Parent ]

  •  Fascinating look. (2+ / 0-)
    Recommended by:
    MichaelNY, bythesea

    Logical that a state where one party significantly outnumbers another may handicap the polls.

    "Be who you are and say what you feel because those who mind don't matter and those who matter don't mind." -Theodore Seuss Geisel

    by KingofSpades on Fri Jun 15, 2012 at 04:36:17 PM PDT

  •  Great analysis. Statistically (5+ / 0-)

    literate discussions of polling are few and far between here.

    Forgive me for a moment, though, as I lapse into my usual schtick. While I appreciate your ongoing series on sources of polling error and look forward to future installments, I'd like to throw in the caveat that, if you step back and look at the big picture, polling averages are pretty damn accurate.

    Granted, such an assessment is in the eye of the beholder -- but I'll use as the baseline the assumptions that readers bring into the discussion. While there is some danger that some folks will credulously buy the results of any poll, the bigger danger I see among dKos commenters is the wholesale dismissal of polling as being a "worthless" endeavor. Everything from, "they didn't call me, so how can it be accurate?" to "polling firms fudge the data in order to spin an ideologically favorable narrative." So to me, it's worth mentioning that blanket dismissal is farther from the empirical reality of the endeavor than blanket acceptance. It's analogous to the complaint by airlines that the news covers every single plane crash in the world, but we don't hear about all of the safe landings. As your charts show, safe landings are far more common in this domain too.

    You are reading my signature line. #hashtag

    by cardinal on Fri Jun 15, 2012 at 05:12:14 PM PDT

    •  Oh, and after all (5+ / 0-)

      that, I forgot to make my other point: your correlation between state partisanship and polling error is really freakin' cool.

      You are reading my signature line. #hashtag

      by cardinal on Fri Jun 15, 2012 at 05:16:58 PM PDT

      [ Parent ]

    •  Thank you (7+ / 0-)

      Thanks for your usual schtick.  It is a useful reminder.  I write from the perspective that polling is generally useful, of course, but we would like to know when there are mistakes, and it's interesting to see how the mistakes happen.  But I will try to make my perspective more explicit in my text; I see I could have done a better job here.

      (Quickly checks what has been written so far of the next diary.....)

      Ok, so the second sentence in my next diary is "Basically polling correctly predicted the winner of every swing state except Indiana, which was close."  Looks OK.  But I will keep your comment in mind.  Thanks again!

      •  It's always a balancing act. (4+ / 0-)
        Recommended by:
        MichaelNY, bythesea, DemFromCT, Mokurai

        When I teach undergrads about polling, I start on a positive note -- "despite what you might hear, polling really can work" -- including a visual and relatively non-technical illustration of sampling logic. But then the rest of the lecture is, "however, here are all of the ways it can go wrong, and here are several questions you must ask of any poll before trusting its result. . ."

        You are reading my signature line. #hashtag

        by cardinal on Fri Jun 15, 2012 at 05:33:43 PM PDT

        [ Parent ]

    •  They are (0+ / 0-)

      in close races.  

      I am really concerned about the cell phone issue though, and am trying to analyze where the error would be.

      The bitter truth of deep inequality has been disguised by an era of cheap imported goods and the anyone-can-make-it celebrity myth - Polly Toynbee

      by fladem on Sat Jun 16, 2012 at 09:57:23 AM PDT

      [ Parent ]

  •  Where does Nate Silver have MA? (1+ / 0-)
    Recommended by:
    MichaelNY

    I quit reading him as much after he moved to the NYT.

    •  I'm not sure (0+ / 0-)

      Paywall, you know.  But the ever-trustworthy DK Elections has it listed as a toss-up.

    •  Nate Silver (1+ / 0-)
      Recommended by:
      MichaelNY

      Hasn't done a really thorough analysis of each Senate race yet, but he did a couple overviews of Senate control at large in which he had MA (along with most of the other general tossup states) at 50-50.

      I think that was just a reflection of conventional wisdon and polling. If/when he goes into each race thoroughly, it will probably change.

  •  Warren should be leading by 20 points. (0+ / 0-)

    At least. I do not get "liberal" Massachusetts' crush on Scott Brown.
    How is it that this race is even close?

  •  Except (3+ / 0-)
    Recommended by:
    fladem, HudsonValleyMark, MichaelNY

    Those races you mentioned at the top of the diary are different from Mass-Sen in one major way. Brown is an incumbent and that is worth something.

    Language professors HATE me!

    by Zornorph on Sat Jun 16, 2012 at 08:08:20 AM PDT

    •  I very much agree with you (4+ / 0-)

      and it worth noting the polling on the special election (which is a different animal, I know) did not understate Democratic strength.

      The bitter truth of deep inequality has been disguised by an era of cheap imported goods and the anyone-can-make-it celebrity myth - Polly Toynbee

      by fladem on Sat Jun 16, 2012 at 10:16:05 AM PDT

      [ Parent ]

      •  I'm not so sure (1+ / 0-)
        Recommended by:
        MichaelNY

        PPP pretty much nailed the result. But I see four polls that ran through January 17, not including R2K; they reported Brown margins of +5, +7, +9, and +10 -- an average of 7.75. Brown won by about 4.8.

        That said, the point about incumbency is important (and the diarist made the same point in a comment yesterday, which I didn't notice at the time). FWIW, the first contest mentioned in the diary was for an open seat. But it would be good to look at that variable systematically.

    •  Hmm, little apparent incumbent effect! (0+ / 0-)

      So I just checked for an incumbent effect and found too few cases to really tell (N=13), unless it's a clear, large effect.  But, hey, let's just peek anyways.

      The margin in polls is overstated in favor of the incumbent by about 3/4 of point - diddle squat.  In other words,  the effect is slightly ANTI-incumbent in nature over the period 2004-2010.  Actual voters are less likely to vote for the incumbent than polls say (throw the bums out).  Warning: the real nature of this effect probably varies by year and party of incumbent.  

      The real power of being an incumbent is 1) quality candidates are less likely to run against you and 2) whoever runs against you, quality or not, is less likely to be able to raise money.  Once you account for money (in House races, at least), the advantage of being an incumbent is only 1 or 2 points.

  •  On the other hand... (1+ / 0-)
    Recommended by:
    MichaelNY

    This effect could and will hurt us in several of our races in red territory--Missouri, Montana, North Dakota. It could possibly even cost us the race in a state like ND, where Heitkamp is leading the polls we've seen but narrowly enough that your effect could pull Berg over the finish line. (Montana I think we are slightly more likely to win and Missouri polls don't show McCaskill leading substantially at this point anyway.)

    •  MO, MT are clear swing states; ND close (0+ / 0-)

      In MO, Obama got 49%; MT, 47%; ND, 45%.  If you look at the graph, the effect of state partisan leaning, on average, should be zilch for MO, one point for MT, and two for ND.  The range of expected departure from the polling margin should be about +5 to -5 for MO, stepping lower for the other states.

      So yeah, Heitkamp needs to lead by 2 in the polling average to have a 50/50 chance, and 7 to feel really comfortable.  But in Missouri, this effect won't come into play, although the other error inherent in polling mean the Dem should be ahead by 5 before we could feel comfortable.

      This year we just happen to lack any close races in seriously red states where the effect would be really noticeable.

      •  MO was about R+7 in 2008 (0+ / 0-)

        So why do you think the state has no R partisan leaning?

        Formerly Pan on Swing State Project

        by MichaelNY on Sun Jun 17, 2012 at 08:25:57 PM PDT

        [ Parent ]

        •  Well... (0+ / 0-)

          I said the effect of the state partisan leaning in MO should be zero, but it also looks like the partisan lean itself is close to zero.  I'm not sure where you get R +7; Cook PVI shows R +3, which in 2008 was about a point away from an exact R/D balance.   The 2008 election itself of course was close to a tie in Missouri, 49 Obama, 49 McCain.

          •  Exactly (0+ / 0-)
            The 2008 election itself of course was close to a tie in Missouri, 49 Obama, 49 McCain.
            In an election in which Obama won nationally by about 7 points. Ergo, MO was 7 points more Republican than the country in 2008. That's not a partisan lean of "close to zero."

            Formerly Pan on Swing State Project

            by MichaelNY on Mon Jun 18, 2012 at 11:29:28 AM PDT

            [ Parent ]

            •  I see what you're saying now. (0+ / 0-)

              You are assuming 'partisan lean' without modifier is comparative partisan lean, I am assuming 'partisan lean' without a modifier is absolute partisan lean.

              So I will try to always use a modifier in the future for clarity!

              •  I would suggest to you (0+ / 0-)

                that disregarding the median will confuse the issue greatly. It would have meant that every state except Minnesota was Republican-leaning in 1984, which is patently untrue, given that the median state would have had something like 13% margin for Reagan.

                And now, getting back to the question of the effect of a partisan lean: How much of a lean do you think is necessary to have a partisan effect? I would say without much hesitation that if McCaskill loses to one of the bozos running on the Republican side, it will be strictly because of a strong Republican lean in Missouri. If the same candidates were running in Wisconsin or Iowa, McCaskill would probably win by at least 10.

                A further remark would be that if things go so badly for Romney and the economy has a sudden upturn, such that Obama loses by no more than 2% or so in Missouri, McCaskill will probably win. But in that case, Obama will be winning by plenty - probably at least all the states he won in 2008 except Indiana, plus Arizona.

                Formerly Pan on Swing State Project

                by MichaelNY on Mon Jun 18, 2012 at 04:14:17 PM PDT

                [ Parent ]

                •  The effect (1+ / 0-)
                  Recommended by:
                  MichaelNY

                  I assume you are referring to the effect discussed in this diary?  That is, that the difference between the margin in the polling average and the actual results is a function of the partisan lean of the state in close races.  

                  You can see how much of a partisan lean is necessary to see this effect in the second graph of the diary.

                  At Obama = 50% (absolute margin =0, relative margin R+7) the effect is zero; this is where the purple line crosses the x-axis.  

                  A point or two away from 50%, the effect is still very small and not likely to be noticeable, especially compared to other noise.  

                  At Obama = 53% (absolute margin D+7, relative margin=0), the effect, on average is that the election margin is +1 for the Democrat compared to the polls.  But because of other noise, the range we would expect to see for an individual contest is from D+6 to R+4.

                  At Obama = 57% (absolute margin D+14, relative margin D+7) the effect, on average is that the election margin is +3 for the Democrat compared to the polls.  But because of other noise, the range we would expect to see for an individual contest is from D+8 to R+2.  This is getting more noticeable.

                  At Obama = 62% (absolute margin D+22, relative margin D+15) the effect, on average is that the election margin is +5 for the Democrat compared to the polls.  But because of other noise, the range we would expect to see for an individual contest is from D+10 to D+1.  This is the point, in deep blue states, where an effect has been noticeable 95% of the time in the past.

                  The effect goes the other way too; switch out parties in the above descriptions and you're got the right numbers for the magnitude of the effect.

        •  To be clear (1+ / 0-)
          Recommended by:
          MichaelNY

          Cook's system goes by % received, not margin. so if McCain's Missouri % was 3 points higher than his national % (it was something like that?) R+3 is right.

          Male, currently staying in Osaka-01. Voting in RI-01, went to college in IL-01.

          by sapelcovits on Tue Jun 19, 2012 at 03:29:51 AM PDT

          [ Parent ]

          •  Who got the balance of the 7% (0+ / 0-)

            margin between Obama's narrow loss in MO and his average winning margin, nationally?

            Formerly Pan on Swing State Project

            by MichaelNY on Tue Jun 19, 2012 at 03:53:11 AM PDT

            [ Parent ]

            •  not sure what you're asking? n/t (0+ / 0-)

              Male, currently staying in Osaka-01. Voting in RI-01, went to college in IL-01.

              by sapelcovits on Tue Jun 19, 2012 at 03:57:13 AM PDT

              [ Parent ]

              •  Obama won by about 7% nationally (0+ / 0-)

                in 2008. He lost narrowly in MO. Therefore, the difference between his margin of defeat in MO and his margin of victory nationwide was 7%. You're saying that McCain got 3% more in MO than he did nationwide.

                Wait, I think I figured it out. 3% more for McCain and 4% less for Obama would amount to 7% off the margin. So I think I answered my own question. According to Wikipedia: United States presidential election, 2008, Obama got 52.9% and McCain got 45.7% nationally. If we add 3 to McCain's total, it would be 48.7%, and if we subtract 4 from Obama's total, we get 48.9%.

                But, from further down in the cited article, actual election results in MO were 49.29% for Obama and 49.43% for McCain, so McCain got almost 4% more than his average percentage nationally, and Obama got less than 4% less.

                My guess is that the R+3 rating was, as usual, a composite of the last two presidential elections. That's what Cook always does to get their PVIs, isn't it?

                Formerly Pan on Swing State Project

                by MichaelNY on Tue Jun 19, 2012 at 04:23:38 AM PDT

                [ Parent ]

  •  You didn't even... (0+ / 0-)

    ..mention the most infamous Daily Kos polling flop of all time:

    When this website, over & over & over, keep supplying us with polls showing Bill Halter was going to crush Blanche Lincoln.

    After that, I have never been able to take any poll I see here seriously.

    When Fascism comes to America it will be wrapped in excess body fat and carrying a misspelled sign.

    by wyvern on Sat Jun 16, 2012 at 02:41:27 PM PDT

    •  Well obviously (1+ / 0-)
      Recommended by:
      MichaelNY

      because those polls came from R2K, which is a fraudulent pollster. And DK dumped R2K as soon as they were exposed. I'm sorry but that's easily the most illogical reason I've ever seen to not trust polls on here.

      Male, currently staying in Osaka-01. Voting in RI-01, went to college in IL-01.

      by sapelcovits on Sat Jun 16, 2012 at 03:41:10 PM PDT

      [ Parent ]

      •  Thanks for.. (0+ / 0-)

        ..making my point:

        Daily Kos was in bed with a fraudulent pollster.

        A fraudulent pollster that was conveniently pimping pro-Daily Kos propaganda.

        That's why I'll never trust another poll I see here.

        When Fascism comes to America it will be wrapped in excess body fat and carrying a misspelled sign.

        by wyvern on Sat Jun 16, 2012 at 03:58:46 PM PDT

        [ Parent ]

        •  That's your choice (1+ / 0-)
          Recommended by:
          MichaelNY

          but it would not be one well made. Kos did not know Research 2000 was fraudulent at the time - no one did. Once he found out, he immediately dumped them and threatened legal action.

          Male, currently staying in Osaka-01. Voting in RI-01, went to college in IL-01.

          by sapelcovits on Sat Jun 16, 2012 at 04:38:50 PM PDT

          [ Parent ]

Subscribe or Donate to support Daily Kos.

Click here for the mobile view of the site