Skip to main content

Is one polling company better than another?
Is one more biased?
Is one more accurate?

I examine 5 large polling companies (separating Zogby and Zogby Interactive) and how they did in rating Senate races recently.

Details below the fold

What I did:
  I went to The Pollster and looked at their 'hot' Senate races.  For each, I compared various polling companies' numbers to the average for the last 10.

There are two things you want in such numbers: Low bias and low variance.  Bias means a company favors R or D.  Variance is a measure of how tightly together the numbers are.

I measure bias via the mean - a mean close to 0 indicates little bias. A positive number favors D.  The averages don't sum to 0 bcs. not all polls in the last 10 are included

What did I find?

Gallup         mean  2.5 variance  6.28
Mason Dixon    mean  0   variance  5.55
Rasmussen            0.53          9.76
SUSA                 2             12
Zogby               -1.12          22.12
Zogby interact      -1             10

So, at least for Senate races, the best company is Mason Dixon, in terms of both bias and variance.  Worst is harder to say, but Zogby looks very bad (interestingly, their phone polls are WORSE than interactive) as does Gallup

 

Originally posted to plf515 on Sun Oct 08, 2006 at 06:05 PM PDT.

EMAIL TO A FRIEND X
Your Email has been sent.
You must add at least one tag to this diary before publishing it.

Add keywords that describe this diary. Separate multiple keywords with commas.
Tagging tips - Search For Tags - Browse For Tags

?

More Tagging tips:

A tag is a way to search for this diary. If someone is searching for "Barack Obama," is this a diary they'd be trying to find?

Use a person's full name, without any title. Senator Obama may become President Obama, and Michelle Obama might run for office.

If your diary covers an election or elected official, use election tags, which are generally the state abbreviation followed by the office. CA-01 is the first district House seat. CA-Sen covers both senate races. NY-GOV covers the New York governor's race.

Tags do not compound: that is, "education reform" is a completely different tag from "education". A tag like "reform" alone is probably not meaningful.

Consider if one or more of these tags fits your diary: Civil Rights, Community, Congress, Culture, Economy, Education, Elections, Energy, Environment, Health Care, International, Labor, Law, Media, Meta, National Security, Science, Transportation, or White House. If your diary is specific to a state, consider adding the state (California, Texas, etc). Keep in mind, though, that there are many wonderful and important diaries that don't fit in any of these tags. Don't worry if yours doesn't.

You can add a private note to this diary when hotlisting it:
Are you sure you want to remove this diary from your hotlist?
Are you sure you want to remove your recommendation? You can only recommend a diary once, so you will not be able to re-recommend it afterwards.
Rescue this diary, and add a note:
Are you sure you want to remove this diary from Rescue?
Choose where to republish this diary. The diary will be added to the queue for that group. Publish it from the queue to make it appear.

You must be a member of a group to use this feature.

Add a quick update to your diary without changing the diary itself:
Are you sure you want to remove this diary?
(The diary will be removed from the site and returned to your drafts for further editing.)
(The diary will be removed.)
Are you sure you want to save these changes to the published diary?

Comment Preferences

  •  Tip jar (17+ / 0-)

    Sorry the columns don't line up.....

    Tips, recommends, comments, all welcome

    Republicans believe government is the enemy. When they're in charge, they're right

    by plf515 on Sun Oct 08, 2006 at 06:03:27 PM PDT

  •  what does this mean? (3+ / 0-)
    Recommended by:
    jxg, Rupert, plf515

    Does this mean that the mean is correct? Any variance from the average makes it a poor poll? I don't get it. Go back to '04 and compare the polls to the actual voting results. I'm sure that's been done. How do they compare there?

    •  Well, it may have been done (1+ / 0-)
      Recommended by:
      Rupert

      but I don't have the data handy

      The mean just compares to other polls --- if the whole gang is biased, that won't work, but that seems unlikely

      Republicans believe government is the enemy. When they're in charge, they're right

      by plf515 on Sun Oct 08, 2006 at 06:12:35 PM PDT

      [ Parent ]

  •  Well polls can tell you things (2+ / 0-)
    Recommended by:
    Rupert, plf515

    even if they don't seem all that accurate.  For example, you can spot trends in polls even they're biased in one direction or another.  I noticed that in the California Special Election of 2005.  SUSA polled and showed all the props winning by large margins.  A lot of people pointed out that these polls were biased.  Well we all agreed on that (conflicted with LA Times, PPIC, and Field Polls).  However, in the last week of the campaign, we could see all the polls starting to trend against the propositions and this was seen in the SUSA polls even as they showed a number of Arnold's props still passing.

    •  As long as the bias is consistent (4+ / 0-)
      Recommended by:
      Rupert, SoCalLiberal, wiscmass, blueoasis

      and the variance is low, this is true.

      But if the bias changes over time, then the trend in the poll could be actual change, or change in the bias.  

      And, if the variance is high, then there is a lot of noise, and the trend is hard to spot

      Republicans believe government is the enemy. When they're in charge, they're right

      by plf515 on Sun Oct 08, 2006 at 06:35:02 PM PDT

      [ Parent ]

  •  Possible fraud makes it hard to know (1+ / 0-)
    Recommended by:
    Rupert

    We have respectable figures such as RFK junior providing evidence of fraud, and the vulnerabilities of the voting machines are documented. we will never know which poll is superior.

    •  Actually (2+ / 0-)
      Recommended by:
      Rupert, wiscmass

      low variance is an indicator of qualtiy in a poll, regardless of any fraud in the actual election.

      Fraud would make it hard to tell if the difference between actual results and polls was bias or fraud, but low variance would still be good.

      Republicans believe government is the enemy. When they're in charge, they're right

      by plf515 on Sun Oct 08, 2006 at 06:36:18 PM PDT

      [ Parent ]

  •  Question... (5+ / 0-)
    Recommended by:
    George, jxg, Rupert, blueoasis, plf515

    Did you attempt in some way to account for changes in the mean over time?  Just averaging the last 10 polls might include polls taken weeks or even months apart in some races.  Some times a poll that looks like an outlier is later proved correct by later polling.  It turns out it was not an outlier at all, just the first poll to show a new trend.

    A perfect example is the Gallup poll in the RI race a week ago that showed an 11 point lead for Whitehouse when all other polls had been showing about a 4 point lead.  I was told that poll was an outlier and not worth reporting or relying on by a few people.  But today Rasmussen has a poll showing Whitehouse up by 9.  So now the Gallup poll seems alot more credible.  But if you averaged them both with the previous, much lower, 8 polls to get an average they would seem to be the ones with the greatest variance and bias, where in reality they were just the first ones to catch the aftermath of the Foley fallout in RI.

    If faced with a stark choice between security and freedom, doesn't it seem like the choice of a courageous, patriotic American should be obvious?

    by Ken in Tex on Sun Oct 08, 2006 at 06:28:31 PM PDT

    •  Bingo (2+ / 0-)
      Recommended by:
      Rupert, plf515

      Averaging a poll from May with a poll from last week and then comparing to Pollster's X's numbers from August is meaningless. plf515, I enjoy your diaries, but I think you whiffed on this one.

      "I believe that withdrawal is now the more prudent option." Kay Bailey Hutchinson - (1993)

      by George on Sun Oct 08, 2006 at 06:34:45 PM PDT

      [ Parent ]

      •  Well it can be done (3+ / 0-)
        Recommended by:
        George, Rupert, plf515

        But the math is a more complex.  It involves establishing averaged trend lines that vary over time, and are retroactively updated when newer polls prove out trends.  But that requires a ton of data points.  It's the method that is used at the Pollkatz site.  He does it with 15 different pollsters for Presidential approval ratings.  He gets 5 or more data points a week, so it's possible to pick the outliers from the ones showing early trends.

        But for all that, my gut tells me that the finding above are not so inaccurate.  If I were asked (with no data other than my observations for the last 4 months to list those pollsters in order of how accurate their polls were I'd list them in this order:

        Mason Dixon
        Gallup
        Rasmussen
        SUSA
        Zogby

        So even without correcting for time differential, the analysis seems to have come to close to the right answer.  Might be a just a quirk, or might not.

        If faced with a stark choice between security and freedom, doesn't it seem like the choice of a courageous, patriotic American should be obvious?

        by Ken in Tex on Sun Oct 08, 2006 at 06:47:51 PM PDT

        [ Parent ]

        •  It's REALLY hard (2+ / 0-)
          Recommended by:
          George, Rupert

          to do this really well.

          It's really a three stage hierarchical model, with polls nested within both races and times, and likely requiring both random intercepts and random slopes.

          If the data were available in some nice neat form, I might give it a shot --- but I don't know of them in that format --- ASCII or something similar with

          date race  company  result

          would be a minimum

          Republicans believe government is the enemy. When they're in charge, they're right

          by plf515 on Sun Oct 08, 2006 at 06:54:08 PM PDT

          [ Parent ]

    •  The reason I chose 'hot' Senate races (2+ / 0-)
      Recommended by:
      Rupert, blueoasis

      is to avoid just this problem

      Most of these races are polled very often, so the last 10 doesn't cover that long a period.  Of course, this doesn't ENTIRELY eliminate the problem, but it is further ameliorated by the fact that the polls were in different temporal orders in the different states.

      Republicans believe government is the enemy. When they're in charge, they're right

      by plf515 on Sun Oct 08, 2006 at 06:38:01 PM PDT

      [ Parent ]

      •  Ahhh. (2+ / 0-)
        Recommended by:
        Rupert, plf515

        OK, that explains how your data did seem to get them in the same order that my fuzy math intuition would have put them.

        It's a bit more work but perhaps even more descriptive is a plot of the various polls vs. time, with the floating average drawn in.  Make each pollster a different point symbol.  Then take a step back and look at it.  The outliers and bias really jump out at you.  Your eyes and brain can do a damn good job of correcting for time and trends, etc.

        Look at Pollkatz's plot of the Pres poll ratings:

        http://www.pollkatz.homestead.com/...

        See how the pink diamonds (Gallup) are always on the high edge of the field.  Notice how the yellow diamonds (ARG) and blue diamonds (NBC) seem to have the most points way outside the limits of the rest of the data.

        If faced with a stark choice between security and freedom, doesn't it seem like the choice of a courageous, patriotic American should be obvious?

        by Ken in Tex on Sun Oct 08, 2006 at 07:01:02 PM PDT

        [ Parent ]

        •  Yeah (1+ / 0-)
          Recommended by:
          Rupert

          I was going to make a nice graph of these numbers, with parallel boxplots, but then I realized I had entered the data in the wrong format, and got lazy.

          I like your diaries, Ken.  But why no graphs?

          Republicans believe government is the enemy. When they're in charge, they're right

          by plf515 on Sun Oct 08, 2006 at 07:05:11 PM PDT

          [ Parent ]

        •  Blue Diamonds Are Quinnipiac, Not NBC (1+ / 0-)
          Recommended by:
          plf515

          I read your entry and was immediately thrown to the floor. As someone who obsesses over polls, I can tell you that NBC/WSJ is almos uncanny in their being consistently near the middle of the pack, and in their  lack of glaringly large fluctuations from one poll to the other.

          NBC/WSJ is yellow triangles, not blue diamonds.

  •  I think Mason Dixon, Survey USA, Rassm. (2+ / 0-)
    Recommended by:
    Rupert, plf515

    are pretty good.

    The only one I don't trust is Zogby.

  •  You can't judge a pollster's performance (2+ / 0-)
    Recommended by:
    Rupert, plf515

    until after the eelction when you look at the finals polls. In 2002, Gallup's Senate polls were very good. In 2004, Mason-Dixon and Survey USA were on the mark.

    You're one microscopic cog in his catastrophic plan designed and directed by his red right hand

    by RandyMI on Sun Oct 08, 2006 at 07:00:37 PM PDT

    •  Actually you can (0+ / 0-)

      as I have done.  

      You have to make a couple assumptions:
        1.  Overall, the polling companies are not biased (this could be examined for previous races)
        2.  Companies use similar techniques over time and state-to-state (someone probably knows this, but I don't).

      The problems, identified above in comments, are that the polls are nested within both race and time, so that it becomes complicated to do a full analysis.
      But I dealt with the former by using hot Senate races.

      Republicans believe government is the enemy. When they're in charge, they're right

      by plf515 on Sun Oct 08, 2006 at 07:07:58 PM PDT

      [ Parent ]

      •  I agree (2+ / 0-)
        Recommended by:
        Rupert, plf515

        The quality of the polling can be determined indepentently, and in advance, of the final results being measured, by checking each pollsters for bias and variance.  But I'm not sure we have enough data to do it rigoriously for the Senate races.  Looking at which pollster was more accurate in 2002 or 2004 is not very helpful either as staffing, ownership, finaces, and methodology of any given pollster might change over the course of two years.

        One point I'll make is that one thing is clear:  Polls for news organizations (ABC,NBC, CBS, CNN, FOX, USA today, etc.) are more valuable if they are news worthy.  So these polls have a vested interest in showing something new or different than the other polls, and have little consequences for being wrong.  Professional polling groups that poll for subscribers and stategists like Rasmussen, Mason Dixon, etc. have a vested interest in being the most accurate and timely.  I tend to trust them more.

        If faced with a stark choice between security and freedom, doesn't it seem like the choice of a courageous, patriotic American should be obvious?

        by Ken in Tex on Sun Oct 08, 2006 at 07:18:47 PM PDT

        [ Parent ]

    •  Actually in 2004 Rasmussen was most accurate (1+ / 0-)
      Recommended by:
      plf515

      followed by SUSA, Stategic Vision (yes really) and Mason-Dixon, in that order.

  •  that tells me then (3+ / 0-)
    Recommended by:
    Rupert, blueoasis, plf515

    that if gallup has Dems ahead, that means the Dem is probably very likely ahead.

    •  Just the reverse (0+ / 0-)

      Gallup's mean of 2.5 favors Dems, so if Gallup has the Rethugs ahead, they very likely are

      Still, the best thing to do is look at ALL the polls.

      Republicans believe government is the enemy. When they're in charge, they're right

      by plf515 on Mon Oct 09, 2006 at 02:53:33 AM PDT

      [ Parent ]

Subscribe or Donate to support Daily Kos.

Click here for the mobile view of the site