Skip to main content

Roughly two years after it came to light that Research 2000 appeared to be cooking their numbers, I think it's worth evaluating the performance thus far of Public Policy Polling (PPP) as they have taken over polling for dKos. The evaluation isn't any indictment of the ethics or business practices of PPP, but rather an attempt to determine whether PPP's numbers do accurately reflect the state of public opinion. Put more simply, I'd like to learn whether PPP has a "house effect" that favors Democrats and whether that needs to be considered when weighing PPP's results.

Nate Silver and others have documented a house effect and resulting bias in favor of Republican candidates on the part of Rasmussen Reports. Silver defines both as follows:

When I talk about a house effect, I mean what results a pollster shows relative to those of other polling firms. If everyone else shows the Republican Sharron Angle ahead by 3 points in Nevada and some other pollster comes along showing Harry Reid ahead by 5 points, that would be an 8-point Democratic-leaning house effect.

By bias, I mean how a pollster’s results compared with the actual outcome of the election. In the example above — since Mr. Reid won by about 5 points — the pollster showing him winning by that margin would have an unbiased poll, even though it had a strong house effect. In contrast, the pollsters with Ms. Angle ahead would have a strong Republican-leaning bias, even though they hadn’t had any house effect.

Put another way: house effects are what we look at before the election; bias is what we look at after the election.

Ultimately a house effect is immaterial if there is no bias in the ultimate result. Meaning, if a pollster with a Democratic or Republican house effect turns out to be right, the house effect (which is measured relative to other pollsters) is meaningless. So, I think we need to look at the PPP results to see if there is a bias. PPP's polling with the most recent state and federal elections look like this:
Election PPP Predicted Spread Result Bias
AZ-08 Special D+12 D+7 D+5
WI Recall R+3 R+7 D+4
NY-26 Special D+6 D+4 D+2
MS-GOV R+14 R+22 D+8
NV-02 Special R+13 R+22 D+9
NY-09 Special R+6 R+5 R+1
WV-GOV Special D+1 D+3 R+2
KY-GOV D+27 D+20 D+7

To be sure, there are some variations. In two of the races, PPP actually understated Democratic performance. And in the KY-GOV race, the last poll of theirs that I could find was probably too far in advance of the election to give it the same weight as the others. But generally, PPP has overstated Democratic performance. Moreover, NV-02 ignores that only three weeks before their final poll, they had the eventual winner, Republican Mark Amodei leading only by one. While three weeks can be an eternity in politics, very little occurred in those three weeks to spur a 21-point change.

At the same time, special elections can be tough to gauge--generally more so than primaries or general elections. So this shouldn't be seen as an indictment of PPP's methodology per se, but I do think that they merit watching to see if they're moving in the direction of a Democratic Rasmussen.

EMAIL TO A FRIEND X
Your Email has been sent.
You must add at least one tag to this diary before publishing it.

Add keywords that describe this diary. Separate multiple keywords with commas.
Tagging tips - Search For Tags - Browse For Tags

?

More Tagging tips:

A tag is a way to search for this diary. If someone is searching for "Barack Obama," is this a diary they'd be trying to find?

Use a person's full name, without any title. Senator Obama may become President Obama, and Michelle Obama might run for office.

If your diary covers an election or elected official, use election tags, which are generally the state abbreviation followed by the office. CA-01 is the first district House seat. CA-Sen covers both senate races. NY-GOV covers the New York governor's race.

Tags do not compound: that is, "education reform" is a completely different tag from "education". A tag like "reform" alone is probably not meaningful.

Consider if one or more of these tags fits your diary: Civil Rights, Community, Congress, Culture, Economy, Education, Elections, Energy, Environment, Health Care, International, Labor, Law, Media, Meta, National Security, Science, Transportation, or White House. If your diary is specific to a state, consider adding the state (California, Texas, etc). Keep in mind, though, that there are many wonderful and important diaries that don't fit in any of these tags. Don't worry if yours doesn't.

You can add a private note to this diary when hotlisting it:
Are you sure you want to remove this diary from your hotlist?
Are you sure you want to remove your recommendation? You can only recommend a diary once, so you will not be able to re-recommend it afterwards.
Rescue this diary, and add a note:
Are you sure you want to remove this diary from Rescue?
Choose where to republish this diary. The diary will be added to the queue for that group. Publish it from the queue to make it appear.

You must be a member of a group to use this feature.

Add a quick update to your diary without changing the diary itself:
Are you sure you want to remove this diary?
(The diary will be removed from the site and returned to your drafts for further editing.)
(The diary will be removed.)
Are you sure you want to save these changes to the published diary?

Comment Preferences

  •  Propinas, pourboires, tips (4+ / 0-)

    Unapologetic Obama supporter.

    by Red Sox on Thu Jun 14, 2012 at 07:54:11 AM PDT

  •  In 2010, they were the most accurate pollster. (1+ / 0-)
    Recommended by:
    Supavash

    Consistently.

    "Nothing in all the world is more dangerous than sincere ignorance and conscientious stupidity." --M. L. King "You can't fix stupid" --Ron White -6.00, -5.18

    by zenbassoon on Thu Jun 14, 2012 at 07:56:59 AM PDT

    •  I keep reading that (0+ / 0-)

      assertion here; but do you have a specific citation? The studies I've seen of 2010 contradict that. Here and here, for example.

      You are reading my signature line. #hashtag

      by cardinal on Thu Jun 14, 2012 at 08:41:17 AM PDT

      [ Parent ]

      •  I would like to see the same. (1+ / 0-)
        Recommended by:
        cardinal
      •  Zenbasson overstate's the facts (0+ / 0-)

        PPP was not the best in 2010.

        However, the 538 study shows that PPP had the least house effect.  And the ARG study shows that PPP's accuracy was equal to the Real Clear Politics average, the 538 forecast and the Pollster.com average.

        So, PPP did very well in 2010.

        As "Scarce" says in a post below, any study of PPP's 2012 accuracy would have to include all of their primary and special elections polls this cycle, and I would add that the diarist should have compared PPP's results without those of other pollsters.

  •  Do some real work and get a larger sample size (1+ / 0-)
    Recommended by:
    Supavash

    PPP have probably done a hundred polls this year. Exert yourself and see where they're really at. At least some commenters here had noticed a republican tilt to their results, prior to Wisconsin anyway.

  •  statistical bias is a bit more complicated... (0+ / 0-)

    ...than the definition being used.

    Statistical Bias
    Characteristics of an experimental or sampling design, or the mathematical treatment of data, that systematically affects the results of a study so as to produce incorrect, unjustified, or inappropriate inferences or conclusions.
    Cheers.
  •  OK, two things to consider (0+ / 0-)

    1) Special elections have more error
    2) Blow-out elections are usually the result of one side not showing up. This in turn means everyone's likely voter model winds up wrong. De-weight MS-Gov, KY-Gov, and NV-02 accordingly.

    In theory, there is no difference between theory and practice; but in practice, there always is a difference. - Yogi Berra En théorie, il n'y a aucune différence entre théorie et pratique, mais en pratique, il y a toujours une différence. - Yogi Berra

    by blue aardvark on Thu Jun 14, 2012 at 08:09:56 AM PDT

  •  Good start, though it's not (1+ / 0-)
    Recommended by:
    PatriciaVa

    particularly informative to look at the bias of one pollster in isolation. How did they compare to other pollsters who polled those races?

    You are reading my signature line. #hashtag

    by cardinal on Thu Jun 14, 2012 at 08:28:21 AM PDT

  •  PPP polls races that no one wants to touch. (0+ / 0-)

    In AZ 8, they gave us valuable information that the race was not a toss-up and they were right. Even thus narrow list hows their penchant or accuracy.  Just look at the 2 NY races.  Was anyone even lose on both? Also we're they actually wrong about a result?  They seem like a reliable pollster to me.

    Alternative rock with something to say: http://www.myspace.com/globalshakedown

    by khyber900 on Thu Jun 14, 2012 at 09:07:41 AM PDT

Subscribe or Donate to support Daily Kos.

Click here for the mobile view of the site