While Pew is an experienced and well-respected polling firm, I’d like to make a couple of points about why today’s poll may be fatally flawed by no fault of Pew’s other than the timing of the survey. What I’m going to focus on is the almost irrefutable proposition that this survey was conducted during a time of unusually high variability in the electorate’s perceptions of the race, and the consequences of that variability on the reliability of the data.
Pew was in the field for four days, from Thursday through Sunday, Oct. 4 through Oct. 7 – quite possibly the most volatile four days in the entire election cycle. We have at least three reasonably reliable data points that suggest that Romney received a significant bump from Wednesday’s debate that began to register with polling on Friday, and that began to recede significantly beginning on either Saturday or Sunday. Both the Gallup and Rasmussen tracking polls moved toward Romney beginning with surveys that took place on Friday. Gallup showed Obama going from +5 to +3, a two point move toward Romney, beginning with their poll that included Friday. That same spread continued with their polling that included Saturday. But with Sunday’s polling, Obama recovered his losses as Gallup reverted back to a +5 spread. Rasmussen went from +2 Obama to +2 Romney, a four point swing toward Romney, also beginning with their poll that included Friday. That spread continued with their polling that included Saturday. But with their poll that included Sunday, Obama had pulled even, regaining two of the four points he lost on the previous two days. Gallup uses a seven day rolling average. That means with each new poll, the poll that is eight days old falls of the back end. Rasmussen uses a three day rolling average. In order to counter the effect of two days of bad polling in a single day on a rolling average, the single day is likely to have been better than the previous two days were worse. The Gallup poll’s response to the debate was likely muted compared to Rasmussen’s because their data is averaged over a longer period.
Both Gallup and Rasmussen are showing significant improvement for Obama from the three days immediately following the debate. Adding one more data point, PPP was in the field in Virginia during this period and tweeted, “Friday was a strong day for Romney in our VA polling, Thursday, Saturday, and Sunday were about par for the course of our work there this yr.”
Back to Pew. Just about any photographer will tell you if you want to take a sharp picture of a quickly moving object, you need a fast shutter speed. Multiple day polls are generally conducted with the expectation that the data is relatively static during the polling period, and this assumption is built into the stated accuracy of the poll. When the data is highly volatile within the period of the poll, not only will the results be fuzzy, the most important predictive factor of the poll – the state of the race today - can't be determined from the published data. For these reasons alone, I think this poll can be taken with a jar of Himalayan Pink.
As far as party weighting, I have to say I’m surprised to see so many people making essentially the same argument against this Pew poll that has been made by the unskewedpolls.com people against just about every pre-debate poll ever taken, and that was dismissed with appropriate ridicule by the smart people who do real polling. Pew, like most reputable pollsters, doesn’t artificially weight party representation. If they call 100 people and 30 of them identify as Republicans, then 30% of their respondents are Republicans. It’s as simple as that, and it’s how good polling outfits do it. Party identification is considered fluid. If you don’t like the party weighting you see in a poll, have another debate and take another poll.
As far as I know, the only weighting that gets done by reputable polling firms is by demographic groups like gender, age, and race. That kind of weighting is necessary in order to compensate for the relatively small sample sizes in most polls, and it can have a significant effect on the bottom line. There are scenarios where this kind of weighting could have an exaggerated effect on a poll that was taken on successive days where the data was moving from day to day, but the weighting was done in aggregate. This kind of error could work in either direction, but the fact that Pew is reporting Obama and Romney even with women makes me at least somewhat suspicious. One thing we do know is that the daily tracking polls, by definition, weight their demographics daily, and that could be another reason this poll appears divergent from what we’re seeing in the trackers.
The only definitive poll happens on November 6, and I think this poll is sketchy enough that there isn't any reason to lose sleep over it.