Stochastic Democracy looks at Micheal Weissman's Statistical Critique of Former Kos-Pollster R2K (Research-2000) and finds them it somewhat questionable and not terribly convinving.
See more below the fold
The news of the week is that, on the heels of an investigative critique by Micheal Weissman, Kos is suing their former Pollster R2K (Research-2000) for allegedly falsifying their polls.
This is the same team who looked at Strategic Vision with Fourier analysis. Strategic Vision polls were probably falsified, but Weissman's Fourier Analysis technique wasn't very robust, rejecting Quinnipac as a real pollster with 95% confidence.
So to start, the paper's team consists of a retired physicist , A Wildlife research technician, and a political consultant. That doesn't necessarily discount their work, but polling stratification is very complex, and I wouldn't even necessarily trust Statisticians unless they have specifically worked in the field (Think Andrew Gelman or Doug Rivers).
Getting into the nuts and bolts of things, I don't think these critiques are very plausible. To go through them:
- That the parity(Even/Oddness) of Male and Female cross-tabs always match-
Polls taken of different groups of people may reflect broadly similar
opinions but should not show any detailed connections between minor random details. Let's look at a little sample of R2K's recent results for men (M) and women (F).
6/3/10 Favorable Unfavorable Undecided
Question Men Women Men Women Men Women
Obama 43 59 54 34 3 7
Pelosi 22 52 66 38 12 10
Reid 28 36 60 54 12 10
McConnell 31 17 50 70 19 13
Boehner 26 16 51 67 33 17
Cong. (D) 28 44 64 54 8 2
Cong. (R) 31 13 58 74 11 13
Party (D) 31 45 64 46 5 9
Party (R) 38 20 57 71 5 9
I've seen a good theory that this was caused by crappy integer handling. But crappy integer handling is pretty common and not indicative of fraud.
- That cross-tabs don't fluctuate as much as you'd expect from ideal random sampling (This was also pointed out by Nate Silver)
Pollsters don't use ideal random sampling, and so pulling out a Chi-Squared test to show that their polls could not have come from ideal random sampling does not prove anything.
In reality, Pollsters tend to use complex poll stratification techniques in order make their polls more accurate. Stratified Sampling takes advantage of outside information like Census data or previous polls in order to decrease variability.
Because of the widespread use of Stratified Sampling, a Poll's real margin of error is only loosely related to it's sample-size, and not according to the Stats 101 formulas that Weissman assumes. This is even more true when looking at the margins of error for Cross-tab estimates.
R2K ran a tracker, taking polls frequently at a regular interval. It's not inconceivable that they used information from their previous polls in order to more accurately estimate their cross-tabs. This would be good statistical practice! This, along with numerous other techniques, would produce the same observed effect.
- That the day to day distribution of changes contains too few zeros compared to Gallup -
- This initially seemed strange, but courtesy of
Harry Enten of Pollster.Com, here is a graph of the week to week changes of Gallup's
Disapproval polls:
This isn't as large a disparity as R2K, but it's still larger then would be expected from chance. It adds credence to the idea that poll stratification might have some sort of non-trivial effect that could explain the disparity.
It's certainly possible that R2K falsified polls, especially given Ali's erratic behavior after being accussed, but these methods are not very convincing. If this is all Kos is relying on, I have trouble seeing how they can win their lawsuit.
Background: I'm 18-year old visiting Graduate Student at Princeton University doing statistical analysis with Professor Wang in Neurology and Election Forecasting.
Update: Just to make things clear, I support Kos's decision to dump R2K as a pollster. They seem not to have been terribly competent and had an enormous pro-democratic house-effect. I'm just cautioning that not much weight should be placed on these particular points.