Roughly two years after it came to light that Research 2000 appeared to be cooking their numbers, I think it's worth evaluating the performance thus far of Public Policy Polling (PPP) as they have taken over polling for dKos. The evaluation isn't any indictment of the ethics or business practices of PPP, but rather an attempt to determine whether PPP's numbers do accurately reflect the state of public opinion. Put more simply, I'd like to learn whether PPP has a "house effect" that favors Democrats and whether that needs to be considered when weighing PPP's results.
Nate Silver and others have documented a house effect and resulting bias in favor of Republican candidates on the part of Rasmussen Reports. Silver defines both as follows:
When I talk about a house effect, I mean what results a pollster shows relative to those of other polling firms. If everyone else shows the Republican Sharron Angle ahead by 3 points in Nevada and some other pollster comes along showing Harry Reid ahead by 5 points, that would be an 8-point Democratic-leaning house effect.
By bias, I mean how a pollster’s results compared with the actual outcome of the election. In the example above — since Mr. Reid won by about 5 points — the pollster showing him winning by that margin would have an unbiased poll, even though it had a strong house effect. In contrast, the pollsters with Ms. Angle ahead would have a strong Republican-leaning bias, even though they hadn’t had any house effect.
Put another way: house effects are what we look at before the election; bias is what we look at after the election.
Ultimately a house effect is immaterial if there is no bias in the ultimate result. Meaning, if a pollster with a Democratic or Republican house effect turns out to be right, the house effect (which is measured relative to other pollsters) is meaningless. So, I think we need to look at the PPP results to see if there is a bias. PPP's polling with the most recent state and federal elections look like this:
Election |
PPP Predicted Spread |
Result |
Bias |
AZ-08 Special |
D+12 |
D+7 |
D+5 |
WI Recall |
R+3 |
R+7 |
D+4 |
NY-26 Special |
D+6 |
D+4 |
D+2 |
MS-GOV |
R+14 |
R+22 |
D+8 |
NV-02 Special |
R+13 |
R+22 |
D+9 |
NY-09 Special |
R+6 |
R+5 |
R+1 |
WV-GOV Special |
D+1 |
D+3 |
R+2 |
KY-GOV |
D+27 |
D+20 |
D+7 |
To be sure, there are some variations. In two of the races, PPP actually understated Democratic performance. And in the KY-GOV race, the last poll of theirs that I could find was probably too far in advance of the election to give it the same weight as the others. But generally, PPP has overstated Democratic performance. Moreover, NV-02 ignores that only three weeks before their final poll, they had the eventual winner, Republican Mark Amodei leading only by one. While three weeks can be an eternity in politics, very little occurred in those three weeks to spur a 21-point change.
At the same time, special elections can be tough to gauge--generally more so than primaries or general elections. So this shouldn't be seen as an indictment of PPP's methodology per se, but I do think that they merit watching to see if they're moving in the direction of a Democratic Rasmussen.