Skip to main content

Armchair political analysts (including yours truly) love to play this political parlor game:

As the avalanche of data pours in during the weeks preceding the election, every poll is dutifully parsed. In particular, campaign junkies of all stripes are fond of dismissing certain pollsters as "partisan" or "biased". This, of course, usually accompanies the release of a poll favoring the opposing party of the accuser.

With the election results (almost entirely) in the bank at this point, however, decent conclusions can be drawn about the partisan proclivities of the leading pollsters in the game.

Some might note that this has already been done, predictably by politics-and-numbers guy Nate Silver.

The concept is the same (determining whether pollsters had their thumbs on the scale towards one party or the other), but the methodologies between Nate's study and this one are quite different. What Silver did was take the number of percentage points, on average, that the pollsters results from the last 21 days of the cycle deviated from the actual results.

There's nothing wrong with this particular method. What follows is simply a slightly different way of looking at it. Here, a pollster was given a score of "accurate" if their poll came within 3 points of the final total. If they were greater than 3 points off in the direction of the GOP, that poll was rated as biased towards the Republicans. If they were greater than 3 points off in the direction of the Democrats, that poll was rated as...well, you get the idea.

The rationale for deviating from Nate on methodology was based on a simple concern. Nate's method extracts a steep price for a pollster with a single, but very aberrant, result. For example, a pollster could err on the side of the GOP four out of five times, but if the pollster was way off on the fifth poll (but in the Democratic direction), it could more than offset those GOP-leaning polls.

Here were the parameters for this study:

  • Every poll conducted (by the pollsters in question) from October 1st until Election Day was included for review. This is a big deviation from what Nate did, and it is bound to have some critics. Using multiple polls of the same race is certainly a controversial provision: a pollsters gets dinged extra if they were consistently wrong on a race for which they were prolific. However, by using just the most recent poll in a race, some pollsters (Quinnipiac certainly comes to mind) would be rewarded because early aberrant results would be left uncounted, as long as they snapped back into line with their final polls. The window of time is a bit longer, as well.
  • Only pollsters who offered at least a dozen polls during that time period stretching from October 1st and Election Day were included in this analysis. This narrowed the participating group down to fifteen pollsters.
  • Because it was hard to gauge what party benefitted from errors in a few races with three legitimate leading candidates, those races were omitted from the study. As a result, polls of the gubernatorial races in Maine and Rhode Island, as well as the Senate races in Florida and Alaska, were not included. The Colorado gubernatorial race was left in, however. The justification? By October 1st, Tom Tancredo had already established himself as the de facto choice of GOP voters, despite technically being an Independent.
  • The pollsters' "bias ratings" were taken, quite simply, by looking at the difference between the percentages of polls favoring one party over another (regardless of party). So, if a pollster had 43% of their polls favoring Republicans, and 29% of their polls favoring Democrats, their "bias rating" would be a 14. Percentages were selected instead of raw numbers of polls, so the more prolific pollsters would not be unfairly impacted.

So, how did the pollsters of America do? The results might surprise you. But first, a brief caveat: a pollster leaning to one party or another should not necessarily be construed as an accusation of intentional behavior by said pollster. It could be (and, in fact, is likely to be more often than not) simply the result of a likely voter screen that was either too restrictive, or not restrictive enough. There are also a couple of key factors that help to explain the top two names on this particular list. Look for those in the commentary which follows the list.

With that caveat in mind, here is the list, in order from the most biased pollster to the least biased pollster:

POLLSTER BIAS RATINGS: 2010 ELECTION CYCLE--10/1/10 to 11/1/10 (Number of polls conducted in parentheses)

  1. Merriman River (18)--Bias rating of 83 (83% R 0% D)
  1. Penn Schoen Berland (42)--Bias rating of 57 (67% D 10% R)
  1. Ipsos/Reuters (12)--Bias rating of 50 (58% R 8% D)
  1. Susquehanna Research* (15)--Bias rating of 46 (53% D 7% R)
  1. Siena College (14)--Bias rating of 43 (50% D 7% R)
  1. Rasmussen* (161)--Bias rating of 33 (49% R 16% D)
  1. CNN/Op Research (28)--Bias rating of 32 (46% R 14% D)
  1. Suffolk University (13)--Bias rating of 30 (38% D 8% R)
  1. YouGov (33)--Bias rating of 24 (39% R 15% D)
  1. SurveyUSA (69)--Bias rating of 24 (46% R 22% D)
  1. Quinnipiac (26)--Bias rating of 20 (35% R 15% D)
  1. PPP (59)--Bias rating of 19 (39% R 20% D)
  1. Public Opinion Strategies (22)--Bias rating of 13 (36% D 23% R)
  1. Mason Dixon (28)--Bias rating of 11 (43% D 32% R)
  1. Monmouth University (14)--Bias rating of 8 (29% R 21% D)

(*)--"Pulse Opinion Research" polls conducted for Fox News were included here, because that firm is affiliated with Rasmussen. The same type of affiliate relationship existed for the work that Susquehanna did for the Sunshine State News in Florida.

A few observations/takeaways from the list:

  • Merriman River shouldn't necessarily be wholly acquitted for pacing the field, but there is a legitimate explanation for their gaudy numbers here. The pollster only did work in two states: Connecticut and Hawaii. Those happened to be two (of the only) states in the Union where the GOP badly underperformed. That doesn't totally excuse their numbers, however: they did have Republican Sam Caligiuri leading Democratic Congressman Chris Murphy by eight points on Election Eve. Murphy wound up winning re-election by over eight points.
  • House races are notoriously difficult to poll, and so that partially acquits another pollster on the medal stand. Penn Schoen Berland only polled House races during this cycle, which might explain their performance a bit. Other pollsters near the top (Ipsos, Rasmussen) polled nothing but statewide races, so they have less of an excuse.
  • The most amazing performance here may well be from Public Opinion Strategies. Kossacks who regularly digested the nightly Polling Wraps routinely read comments dismissing the GOP pollster by their initials (P.O.S.). But they were not only is in the bottom third on the bias question, they actually erred MORE in the direction of the Democrats than they did in the direction of the GOP. An amazing outcome, given that they are a Republican pollster. Now, P.O.S. did have some whoppers (Charlie Baker up 7 in MA-Gov, Matt Doheny up 14 in NY-23), but they also were the canary in the coalmine on a couple of races (MN-08, AL-02).
  • Our own polling partner, PPP, also landed in the least-biased third of the polling crew for this cycle, and actually were more likely to err in the GOP's direction than in the Democratic direction. Perhaps, given the performances of both PPP and Rasmussen in this polling cycle, the press will finally cease referring to PPP in every report as a "liberal" or "Democratic" pollster. Meanwhile, they might also see fit to, in a long overdue gesture, classify Rasmussen as a "Republican" pollster. They certainly earned it this cycle.

One final set of data to peruse. Let's give a well earned shout-out to the pollsters that managed to get closest to the pin. What follows is the top five list of pollsters who were deemed "accurate" (within 3% in either direction) to the final result most often, as a percentage of their total polls:

  1. Suffolk University--54%
  1. Monmouth University--50%
  1. Quinnipiac--50%
  1. YouGov--45%
  1. Siena College--43%

For those wondering, PPP joined Public Opinion Strategies just outside of the top five, at 41%. Rasmussen was several points back at 35%. Here, as with the bias question, Merriman River came in with the worst performance (17%).

Originally posted to Daily Kos on Sun Nov 21, 2010 at 07:00 PM PST.

Tags

EMAIL TO A FRIEND X
Your Email has been sent.
You must add at least one tag to this diary before publishing it.

Add keywords that describe this diary. Separate multiple keywords with commas.
Tagging tips - Search For Tags - Browse For Tags

?

More Tagging tips:

A tag is a way to search for this diary. If someone is searching for "Barack Obama," is this a diary they'd be trying to find?

Use a person's full name, without any title. Senator Obama may become President Obama, and Michelle Obama might run for office.

If your diary covers an election or elected official, use election tags, which are generally the state abbreviation followed by the office. CA-01 is the first district House seat. CA-Sen covers both senate races. NY-GOV covers the New York governor's race.

Tags do not compound: that is, "education reform" is a completely different tag from "education". A tag like "reform" alone is probably not meaningful.

Consider if one or more of these tags fits your diary: Civil Rights, Community, Congress, Culture, Economy, Education, Elections, Energy, Environment, Health Care, International, Labor, Law, Media, Meta, National Security, Science, Transportation, or White House. If your diary is specific to a state, consider adding the state (California, Texas, etc). Keep in mind, though, that there are many wonderful and important diaries that don't fit in any of these tags. Don't worry if yours doesn't.

You can add a private note to this diary when hotlisting it:
Are you sure you want to remove this diary from your hotlist?
Are you sure you want to remove your recommendation? You can only recommend a diary once, so you will not be able to re-recommend it afterwards.
Rescue this diary, and add a note:
Are you sure you want to remove this diary from Rescue?
Choose where to republish this diary. The diary will be added to the queue for that group. Publish it from the queue to make it appear.

You must be a member of a group to use this feature.

Add a quick update to your diary without changing the diary itself:
Are you sure you want to remove this diary?
(The diary will be removed from the site and returned to your drafts for further editing.)
(The diary will be removed.)
Are you sure you want to save these changes to the published diary?

Comment Preferences

  •  I recently noticed that... (8+ / 0-)

    Rasmussen is no longer featuring quotes from Markos or Nate Silver about how great their polls are.

    :-)

  •  slightly off topic question... (1+ / 0-)
    Recommended by:
    Krush

    I recently commented that white voters had voted in the majority for Christine O'Donnell and Sharon Angle but because most minorities in their races voted against them they lost. Other commentors said I was wrong and that white voters in Delaware and Nevada had voted democratic. I tried to google the truth but couldn't find the facts.

    Can someone point me to the answer or the voter breakdown by race? I'm pretty sure it was in a front page article here on dkos just after the election but I'd like to see the demographic stats. thanks

    •  Here's The Breakdown: (7+ / 0-)

      Not true in Delaware. Coons carried all racial groups, including whites (albeit only by a 51-49 margin):

      Delaware exit polls.

      Alas, in Nevada, 'tis true. Angle carried the white voters of the Silver State by a 52-41 margin. Reid owes his seat to the black and Hispanic voters of his state, with which he built up an insurmountable edge:

      Nevada exit polls.

      "You share your young with the wolves of the nation...
      Theres nothing left til you pray for salvation"
      Black Rebel Motorcycle Club "American X"

      by Steve Singiser on Sun Nov 21, 2010 at 07:37:17 PM PST

      [ Parent ]

      •  thanks, I was halfright n/t (2+ / 0-)
        Recommended by:
        Steve Singiser, Matt Z
      •  That's just frightening. (0+ / 0-)

        Of those 2, Hispanics can't be counted on as Dem voters.  Polls show they tend to vote for the candidate, not the Party, and I think they'll usually vote for an Hispanic candidate (doesn't matter which Party).  They voted for Harry but also for Sandoval (who had no stated budget or plan for going forward) who is in favor of the Az law.  That makes no sense except that he's Hispanic.  There were other races (no pun intended), too, that support my assertion but I don't remember which one's.  New Mexico's Governor, I think.

        "Only Democrats could lose in a popularity contest to someone MORE unpopular than themselves." - Bill Maher

        by rainmanjr on Sun Nov 21, 2010 at 07:48:53 PM PST

        [ Parent ]

        •  My point was about whites, though. (2+ / 0-)
          Recommended by:
          Matt Z, lostinamerica

          White Angle voters is just amazing.  Could be because unions are heavily populated with blacks and Hispanics.  Whites are more likely to be unemployed since fewer of us are union so had less protection.  That's certainly true for casino workers.  That makes Sandoval's win even more attributable to his nationality since unions favored Rory Reid.  My guess is that Hispanics are not as affected by the economic fallout as others because they are very family-supported (with large families) and get more off-the-record jobs.  I don't mean these thoughts racially, only numerically, and have no idea if any polls back up my assertions.

          "Only Democrats could lose in a popularity contest to someone MORE unpopular than themselves." - Bill Maher

          by rainmanjr on Sun Nov 21, 2010 at 08:01:24 PM PST

          [ Parent ]

  •  Any data on the Field Poll? (0+ / 0-)

    Field Research Corporation is considered the gold-standard in the Golden State - did that hold up this cycle?

    The real enemy of the good is not the perfect, but the mediocre.

    by Orange County Liberal on Sun Nov 21, 2010 at 07:25:53 PM PST

  •  Rasmussen ruined his career with his performance (2+ / 0-)
    Recommended by:
    Matt Z, rainmanjr

    In 2010. Of cause the phony media who wants to drive the narrative that the president is doomed in 2012 will continue to run with his numbers. At least serious people have learnt to dismiss is garbage.

  •  It's Siena College (4+ / 0-)
    Recommended by:
    Steve Singiser, Matt Z, rainmanjr, annieli

    I attended this Franciscan institution outside of Albany NY way back when.

    The University of Siena is in Italy.

  •  SurveyUSA did very well, I thought. (0+ / 0-)

    If I'm following correctly, they had a good bias rating for a rather large number of polls.  

    "Only Democrats could lose in a popularity contest to someone MORE unpopular than themselves." - Bill Maher

    by rainmanjr on Sun Nov 21, 2010 at 07:42:41 PM PST

  •  Very Nice Work! (4+ / 0-)
    Recommended by:
    Matt Z, rainmanjr, Govinda, keetz4
    Thanks for that analysis. It sheds a lot of light on where these guys are.

    One thing I would like to see, however, is something that takes into account earlier polls. While it's true that the numbers don't tend to settle until nearer the election, I also think that early polls tend to set a tone for a race, especially national races in well-watched states and districts. Any bias here would be magnified. So, perhaps we need an analysis of early poll results and their relationship to the final outcome.

    Not that I'm suggesting anything!

    •  Might be better to seperate the 2 stages. (3+ / 0-)
      Recommended by:
      Liberal Thinking, Matt Z, keetz4

      1st half polling (till Aug) and 2nd half polling (Sep-Nov) might be interesting.  That would tell us more about any tone the earlier polls might have set, I think.

      "Only Democrats could lose in a popularity contest to someone MORE unpopular than themselves." - Bill Maher

      by rainmanjr on Sun Nov 21, 2010 at 08:04:29 PM PST

      [ Parent ]

  •  As long as it's a parlor game, are there any (0+ / 0-)

    measures/indices of "push" among these pollsters?

    "calling for a 5" deck gun is not parody. Not by a long shot." (gnaborretni),Warning-Some Snark Above

    by annieli on Sun Nov 21, 2010 at 08:03:05 PM PST

  •  Hawaii (0+ / 0-)

    PPP polled the Governors race between Abercrombie and Aiona as even.  Abercrombie won by 17 points.  Other polls during the year were also useless.  

    "It's too LATE to stop now!" - John Lee Hooker

    by Rolfyboy6 on Sun Nov 21, 2010 at 08:14:18 PM PST

  •  Another factor I'd like to see included.... (1+ / 0-)
    Recommended by:
    Matt Z

    Not that I'm telling you what to do or how to live your life, but I'd be interested in seeing another variable included.

    Given that the voting public does bob & sway as election day approaches (and a gazillion dollars USD of attack ads get launched), I'd be interested in seeing how a poll deviates from what others are saying.

    For example, Rasmussen appears almost borderline reasonable (almost) under your methodology, but their polls (so they seemed to me) were consistently three to four points outside the average of all other polls taken at the same time.

    You also noted that Ras, while polling the high & low holy crap out of everything, turned strangely silent in the final week before the election....You suggested that perhaps they were trying not to look the doofoos (sic) with the late breaking boners (sic).

    While I'm bossing you around, were there areas of the country more prone to polling idiocy?  Living in Oregon, most every poll of our Governor's race had the parties within the margin of error, and that election did prove very close.  But I remember seeing polls for the the Reid/ Sharrrrrrron (sic) race that were all over the map....

    •  I agree with this (1+ / 0-)
      Recommended by:
      Bill Abendroth

      The problem with this kind of analysis is that just because a poll done a month before the election gets the results wrong, it doesn't follow that the poll was wrong- it is possible the poll is dead on, and that if the election were held a month earlier, the results would have reflected that.

      The better analysis is really to see, what were all the other pollsters saying at (roughly) the same time.  

      One other complaint- this applies to Nate's analysis as well- 3 points off is not the same in every race.  I would argue a pollster that states candidate A is going to win by 20 points, when in fact he wins by 30, is more "accurate" than the pollster who says candidate A is going to win by five points, and instead he wins by only one.  If a race is predicted to be a blowout, and in fact it is, then it should be tossed from the analyis.

      One other thing- shouldn't the margin of error be factored in as well?  Shouldn't a poll that flat out states it is plus or minus 4 points be held to a lesser standard than one that claims to be only plus or minus 2?

      If there is anything I have learned from Scooby Doo, it is that the only thing to fear is crooked real estate developers.

      by JakeC on Mon Nov 22, 2010 at 10:49:26 AM PST

      [ Parent ]

      •  Jake & I are taking turns... (1+ / 0-)
        Recommended by:
        JakeC

        ...patting each other on the back.

        I hadn't thought about "blow out" elections, that might skew the accuracy of polling (the Alvin Greene--Jim Demint debacle, being the classic).

        I'd be interested in a targeted analysis of the close elections, and who consistently opted for the losing party.  As someone living in Ecotopia, what I remember best about Rassmussen is they almost always had Dino "Saur" Rossi beating Senator Murray.  Or Miller ahead of Murkowski.  

        If you're looking at which pollsters are in fact "push polling," predicting by what magnitude Blanche Lincoln would be beaten like rented mule, is not where to look.......

Subscribe or Donate to support Daily Kos.

Click here for the mobile view of the site