Skip to main content

I got into a heated dispute with a friend recently over the quality of Rasmussen's polling.  I even went so far as to dig up Nate Silver's report produced some time back showing that Rasmussen has about a 3% Republican-favorable bias (and that is in their late polling, which almost always drifts towards reality - who knows how horribly twisted their early polls are).  Well, I've crunched the numbers.  Join me below the fold to see Rasmussen fail again.  Please, share this data with any of your right-leaning poll-junkie friends...

I've compiled all of the errors for pollsters polling several races.  I looked at only statewide races.  And to assure objectivity, I did not want to define which ones to use.  So, I used Real Clear Politics website and analyzed every Senate and Governors race that they deemed a toss-up or a "lean".  The only exceptions were that I ignored the Rhode Island governor and Alaska senate races, since it is difficult to parse them in terms of determining a Republican or Democratic poll bias (note, I also considered Tancredo as the Republican in the Colorado governor's race).

Here's a table of the data:

Untitled - 1

Several observations can be made:

  1.  First, a non-Ras observation... KUDOS to all who participated in the GOTV effort.  Despite taking a serious hit on Tuesday, EVERY pollster's mean polling bias was Republican-friendly compared to actual results.  Considering that undecideds historically go to the challenger, and more Democrats were incumbent than challenger... in THEORY Republicans should have overperformed the polls.  In fact, Democrats, bad as the night was, did BETTER than polls suggested (at least at the Senate and Governor levels).  I attribute this, at least in part, to a better ground game... getting more of our not-likely voters to the polls.  Good job!
  1.  Rasmussen's bias?  Wait for it.  Wait for it.  Wait for it.  Out of a total of 30 races, Rasmussen's bias was 3.0% Republican biased!  Dead on with their historical record.  Note:  For future reference, if you want to look at Rasmussen data, it can actually be VERY useful... just always add 3% to the Democrat's column (or subtract 3% from the Republican).
  1.  Out of 30 races, Rasmussen's polling was too Democratic friendly a grand total of four races (MO-Sen, MN-Gov, VT-Gov and TX-Gov).  They had zero bias in only three races (KY-Sen, GA-Gov and PA-Gov).  They had a Republican bias in the remaining 23 races!! (CT-Sen, CA-Sen, CO-Sen, IL-Sen, NV-Sen, PA-Sen, WA-Sen, WV-Sen, WI-Sen, CA-Gov, NH-Gov, CO-Gov, CT-Gov, FL-Gov, HI-Gov, IL-Gov, MA-Gov, OH-Gov, OR-Gov, AZ-Gov, NM-Gov, SC-Gov and WI-Gov).
  1.  Now, a devil's advocate may say... Well, point #1 may make point #2 less consequential.  Ras was, after all, not an outlier, since I just said all polls leaned Republican, right?  Wrong, a head-to-head with other pollsters (comparing only mutually-polled races) continues to show Rasmussen with a relatively significant Republican bias:

4a) Versus Quinippiac (6 races).  Quin was 0.8% Republican biased.  Rass was 1.4% Republican biased in those same races.

4b) Versus SUSA (7 races).  SUSA had a 0.1% Republican bias.  In those same races Rass was 2.9% Republican biased.

4c) Versus McClatchy/Marist (7 races).  Mc/Mar came in at a rather poor 3.3% Republican biased.  However, that included the abysmal CO-Gov race, in which Ras did even slightly worse... so, Ras's average for these same seven races was 3.6% Republican biased.  That difference is, arguably, within the noise.  Nonetheless, strictly by the numbers, Ras still lags McClatchy/Marist.

4d) Versus Mason-Dixon (7 races).  MD had some notable errors, but was one of few pollsters with an even split between the number of Dem-biased versus Rep-biased polls.  They came in with only a 0.9% Republican bias (driven primarily by their awful polling of the NV-Sen race).  Rasmussen tallied a 1.7% Republican bias in these same races.  These actually happened to be some of Ras's better polled races, but they too got driven down by the NV-Sen race, and the FL-Gov race hurt them relative to MD (which showed a Democratically biased Sink win).

4e) Versus our own PPP (22 races)!!  PPP had some oddly erroneous late-breaking Republican trends, giving them a pretty substantial 2.1% Republican bias.  Nonetheless, the "good guys" still beat the "bad guys", as Ras pulled a 3.1% Republican bias in these 22 races.

  1.  Bottom line from point (4) is that Rasmussen ranks worse than EVERY other pollster that polled multiple state-wide races... at least in terms of bias.
  1.  I didn't make a similar spreadsheet with absolute values (magnitudes) just to measure errors.  However, here's what the average errors for these six pollsters works out to:  Mason-Dixon 4.3%, Rasmussen 4.0%, McClatchy/Marist 3.9%, PPP 3.7%, SUSA 2.7%, Quin 2.2%.
  1.  But again, they don't all have the same sample, so we've got to compare apples-to-apples.  So, here's Ras compared to mutually-polled races from the other pollsters:

7a) Versus Quin:  Ras error = 1.8%,  Quin error = 2.2%
7b) Versus SUSA:  Ras error = 3.4%,  SUSA error = 2.7%
7c) Versus Mc/Mar:  Ras error = 3.5%, McMar error = 3.9%
7d) Versus MD:  Ras error = 4.0%, MD error = 4.3%
7e) Versus PPP:  Ras error = 3.7%, PPP error = 3.7%

  1.  So, in terms of raw error, Rasmussen seems to be okay... right in the middle of the pack.  However, this is not actually a good thing, because it means that it's horrible bias is a systematic (and, likely, intentional) bias.  Why can we conclude this?  Because if their raw error were worse than all of the other pollsters, then their bias could POSSIBLY be attributed to a few extremely bad polls skewing their results.  But this would show up in their error, making them worse than their competition in this regard as well.  They're not.
  1.  You can see how systematic their bias is by comparing their tendency to over-favor Republicans.  Earlier I note that they were Republican-friendly in 23 of their 30 races polled.  That's a 77% rate.  How about the other pollsters?  Quin leaned too Republican 67% of the time.  PPP was too Republican 64% of the time.  SUSA 57% of the time.  Mc/Mar 86% of the time.  MD 43% of the time.  Only McLatchy/Marist had a worse rate than Rasmussen, but that's just a matter of sample... Mc/Mar was Republican-friendly in six-of-seven races, and in those same seven races, so was Rasmussen!  Moreover, in that 7th race at least Mc/Mar was Dem-friendly... Rasmussen just had zero bias in that race (PA-Gov).

Bottom line, I know I'm preaching to the choir here, but I wanted to back this up with some data...  Rasmussen has a systematic, relatively large (for a mean amongst many polls) Republican bias in their polling.  It appears to be consistently (on average) around 3% ...Nate Silver observed this in past races and it held true this year.  Subjectively, we could argue that earlier in the election cycle it's even larger... but that's tough to gauge.  We know, unequivocally, that they have a significant Republican bias in their final polling results.

Other than PPP the other pollsters had a pretty small sample.  So, it's tough to say anything definitive, but... SUSA appears to be one of the best out there.  Quin's not bad either, esp. after laying some eggs in the last election cycle.  McMar has some work to do.  Mason-Dixon is good in terms of showing little bias, but they do seem to have a fair amount of noise/error.  As for our guys, PPP... their performance was a bit underwhelming but, nonetheless, probably well worth the expense.  They polled nearly as many races as Rasmussen, and though their raw error was no better, their bias was a full percentage point better (that's not insignificant when you're talking about 22 polls).

On a final note, to reiterate point #1 ...notice that actual results were more favorable to Dems than ANY pollster showed and contrary to the typical dynamics of undecideds.  GOTV efforts work!!  Any time we are within a few points in the polls, we can pull it out with the ground game.  Remember that when 2012 rolls around and you get bugged to help out with GOTV efforts.  Do it!  It works!

Originally posted to millwx on Fri Nov 05, 2010 at 09:08 AM PDT.

EMAIL TO A FRIEND X
Your Email has been sent.
You must add at least one tag to this diary before publishing it.

Add keywords that describe this diary. Separate multiple keywords with commas.
Tagging tips - Search For Tags - Browse For Tags

?

More Tagging tips:

A tag is a way to search for this diary. If someone is searching for "Barack Obama," is this a diary they'd be trying to find?

Use a person's full name, without any title. Senator Obama may become President Obama, and Michelle Obama might run for office.

If your diary covers an election or elected official, use election tags, which are generally the state abbreviation followed by the office. CA-01 is the first district House seat. CA-Sen covers both senate races. NY-GOV covers the New York governor's race.

Tags do not compound: that is, "education reform" is a completely different tag from "education". A tag like "reform" alone is probably not meaningful.

Consider if one or more of these tags fits your diary: Civil Rights, Community, Congress, Culture, Economy, Education, Elections, Energy, Environment, Health Care, International, Labor, Law, Media, Meta, National Security, Science, Transportation, or White House. If your diary is specific to a state, consider adding the state (California, Texas, etc). Keep in mind, though, that there are many wonderful and important diaries that don't fit in any of these tags. Don't worry if yours doesn't.

You can add a private note to this diary when hotlisting it:
Are you sure you want to remove this diary from your hotlist?
Are you sure you want to remove your recommendation? You can only recommend a diary once, so you will not be able to re-recommend it afterwards.
Rescue this diary, and add a note:
Are you sure you want to remove this diary from Rescue?
Choose where to republish this diary. The diary will be added to the queue for that group. Publish it from the queue to make it appear.

You must be a member of a group to use this feature.

Add a quick update to your diary without changing the diary itself:
Are you sure you want to remove this diary?
(The diary will be removed from the site and returned to your drafts for further editing.)
(The diary will be removed.)
Are you sure you want to save these changes to the published diary?

Comment Preferences

    •  A better analysis would compare (0+ / 0-)

      Rasmussen only against Quin, PPP, etc for the same races that Rasmussen and the other polling group both provide data for.  So there would be a Ras Vs Quin, Rass Vs PPP, etc.  In this way you compare how well two organizations did on the same set of races.

      The most important way to protect the environment is not to have more than one child.

      by nextstep on Fri Nov 05, 2010 at 09:49:13 AM PDT

      [ Parent ]

      •  I did this!! It's in the text below the table (1+ / 0-)
        Recommended by:
        hold tight

        I apologize for the use of exclamation points, but this is the second time someone has said this.  Do folks not know how to read?!?!?!?

        I understand the numbers and the table are more "flashy", but I analyzed each of the pollsters versus Ras in their similarly polled races only.  It's in the points that follow the table.  It's all there.  Both bias and error.  Compared Ras to each of the others in their similarly polled races.  Read.  Read.  Read.  It's all there.

        Read something before you comment on it.  Please.

        Be that self which one truly is. -Soren Kierkegaard

        by millwx on Fri Nov 05, 2010 at 09:54:48 AM PDT

        [ Parent ]

        •  Your presentation problem is that (0+ / 0-)

          your opening paragraph highlights the 3%, as does the table and then the more useful data such as:

          Versus PPP:  Ras error = 3.7%, PPP error = 3.7%

          Is not presented until much later.

          As PPP is the official Kos pollster, you should have had the "Ras error = 3.7%, PPP error = 3.7%" in the lead paragraph.

          The most important way to protect the environment is not to have more than one child.

          by nextstep on Fri Nov 05, 2010 at 10:06:26 AM PDT

          [ Parent ]

  •  Great analysis, thanks! (9+ / 0-)

    Nate Silver covered this again yesterday, and came to many of the same conclusions (Raz bad; Quinn, SUSA good). Here's the link:
    http://fivethirtyeight.blogs.nytimes...

  •  hey millwx! (3+ / 0-)
    Recommended by:
    NotGeorgeWill, millwx, soms

    thanks for the diary - is hurricane season over already?

    "Kossacks are held to a higher standard. Like Hebrew National hot dogs." - blueaardvark

    by louisev on Fri Nov 05, 2010 at 09:18:06 AM PDT

  •  Given the kinds of questions (3+ / 0-)
    Recommended by:
    raincrow, hold tight, NotGeorgeWill

    they ask, I'm not surprised. Good analysis, thanks!

    I was polled by Rasmussen in early September and wrote a little diary about the kinds of questions they asked. I'm not surprised that it results in a Republican bias.

  •  Aren't you missing a rather important column? (0+ / 0-)

    Specifically: Actual votes.

    The Rasmussen bias is not determined by comparing them to to other pollsters. Other pollsters can be biased in the other direction.

    The bias is determined by comparing Rasmussen polls to actual results.

    LG: You know what? You got spunk. MR: Well, Yes... LG: I hate spunk!

    by dinotrac on Fri Nov 05, 2010 at 09:30:10 AM PDT

    •  Perhaps I didn't explain (4+ / 0-)
      Recommended by:
      RichM, hold tight, soms, Benintn

      That's exactly what the table shows... it is the pollster's margin versus the actual votes.

      Sorry if I didn't explain that well, but that's precisely what the table is... Bias compared to actual results.  **NOT** bias versus another pollster (that's irrelevant because, as you note, different pollsters have different biases).

      Sorry for the confusion, but I've displayed precisely what you (correctly) say should be analyzed.

      Be that self which one truly is. -Soren Kierkegaard

      by millwx on Fri Nov 05, 2010 at 09:33:45 AM PDT

      [ Parent ]

      •  Oops! Must not have read it right. (0+ / 0-)

        Let me go back and check that out.

        LG: You know what? You got spunk. MR: Well, Yes... LG: I hate spunk!

        by dinotrac on Fri Nov 05, 2010 at 09:34:52 AM PDT

        [ Parent ]

      •  You still have an apples and oranges problem (1+ / 0-)
        Recommended by:
        nextstep

        because you didn't compare pollsters based on the races they polled.

        The most obvious example is Hawaii. That's -15, or .5 points of the total Rasmussen bias.  However, none of the pollsters you list polled Hawaii, so their bias calculations are not comparable.

        A quick eyeball tells me Rasmussen and PPP are pretty comparable, and, if you'll recall, PPP is the pollster of choice for DK.

        LG: You know what? You got spunk. MR: Well, Yes... LG: I hate spunk!

        by dinotrac on Fri Nov 05, 2010 at 09:42:49 AM PDT

        [ Parent ]

        •  Yes, I did... (0+ / 0-)

          In the points below the table.

          Read.  :)

          Be that self which one truly is. -Soren Kierkegaard

          by millwx on Fri Nov 05, 2010 at 09:44:04 AM PDT

          [ Parent ]

          •  Yikes. Am I ever impatient, but... (0+ / 0-)

            your own (correctly done) math seems to contradict your conclusion, at least if your intent is to single out Ras.

            Ras was actually LESS biased than other pollsters except for SUSA and PPP, and PPP was a tie.

            You might be making a case for some systemic biases affecting all pollsters, such as people without landlines or under-polling of Spanish speaking latinos, but not a specific Rasmussen bias.

            Everybody, for example, got caught with their pants down in Nevada, missing the huge GOTV effort to latinos.

            Still, kind of fun to play with.

            LG: You know what? You got spunk. MR: Well, Yes... LG: I hate spunk!

            by dinotrac on Fri Nov 05, 2010 at 09:54:51 AM PDT

            [ Parent ]

            •  No... bias and mean absolute error are different. (0+ / 0-)

              Point #7 is the error, of which you are speaking.

              Point #4 is the bias.  Ras has a higher bias than EVERY other pollster.

              In fact, the very fact that they are NOT worse in error (point 7), makes their large bias even more problematic.  If a pollster just has a lot of noise, it can have a skewed bias due to a couple of whacked-out race results.  However, that would also hammer their error.  Ras has no such problem with their error.  As such, their bias is most likely a systematic (intentional) bias due to their methodology.  That's bad.

              Be that self which one truly is. -Soren Kierkegaard

              by millwx on Fri Nov 05, 2010 at 10:02:56 AM PDT

              [ Parent ]

        •  Ras polled freaking everyone. (0+ / 0-)

          You'll never find another comparable pollster because Ras was hyper-active.

          Money is not speech. Corporations are not people.

          by Benintn on Fri Nov 05, 2010 at 09:45:21 AM PDT

          [ Parent ]

  •  Rass sucks... (2+ / 0-)
    Recommended by:
    hold tight, soms

    His partisanship has creeped into his electoral polling, which means that we can now ignore everything he puts out.  Polls trade in credibility and he has none.  Oh and Gallups +15 for GOP was a crock of shit too.

    The only thing I will say in Rass's defense is CO-Gov looks like the polster got it completely wrong.  It wasn't even close.  I don't know if that is because of the weird three-way nature of the race or what.

    QE2 will be our Titanic.

    by RichM on Fri Nov 05, 2010 at 09:30:55 AM PDT

  •  excellent analysis--send this to Nate too! (3+ / 0-)
    Recommended by:
    hold tight, millwx, soms

    what remains is the task (difficult, as you note) of trying to quantify the 'extra' Ras bias that often appears prior to the end of the cycle, in what Kos considers their "narrative-setting" mode.  This of course would be the most egregious example of actual malfeasance, ie actual data manipulation for a specific short-term purpose,rather than systematic standing 'house' error which they might variously explain away. For instance their results seems to undergo a fairly well-observed "regression to the mean" late in race, when the value in narrative-setting is overwhelmed by the value protecting their long-term reputation.  Might be some way to observe that regression.

    Any any case, this year at least they never did regress FULLY to the mean, but ended up 3% to the right of it. Hope Rassmussen enjoys his cruise with the gopper gang...

  •  Good analysis (0+ / 0-)

    but it doesn't make any difference. ras will still be used heavily by various media outlets (particularly fox) and legitimized by the Village and again, particularly by fox.  ras' value isn't in correct polls, but setting narratives and depressing Democratic participation (funding, candidates etc.). As long as he remains legitimate (NOT credible, but legitimate, as defined by the conservative media) and as long as he polls far more often than any other outlet can afford to do, he'll remain relevant to American politics.  One of the reasons he's cited so often, despite his republican bias, is because he has the funding to poll so many races and attitudes. Wingnut welfare will assure him the funding to continue his endeavors.

    A learning experience is one of those things that says, 'You know that thing you just did? Don't do that.' Douglas Adams

    by dougymi on Fri Nov 05, 2010 at 09:37:33 AM PDT

  •  better table would have actual vote % in it, to (0+ / 0-)
  •  Please work on the House next. (0+ / 0-)

    This line of analysis is very interesting, well-done, and worth reading.

    Ras was trying to create the news, not report it.  They were trying to create expectations, not report them.

    Money is not speech. Corporations are not people.

    by Benintn on Fri Nov 05, 2010 at 09:44:21 AM PDT

  •  two things (0+ / 0-)
    1. you really can't do this comparison unless for each pollster you compare all the same races only - only races they all polled.  Otherwise you are comparing apples and oranges.
    1. you should compare the results to the stated margin of error.  If it is within the margin of error it is just that, error.  Now if all the error goes one way, there may be something to look at, but technically it is all just error if it is within the margin of error.

Subscribe or Donate to support Daily Kos.

Click here for the mobile view of the site