One of the most high-profile endeavors for former DK diarist and New York Times blogger Nate Silver has been his use of statistics to "grade" the myriad of pollsters that offer their wares during election season.
Our own polling partners, Public Policy Polling (PPP), as they have historically done, acquitted themselves well in the preliminary 2010 analysis. Their average "error" (polling result vs. actual outcome) was in the top half, and might have placed in the top two, were it not for the fact that PPP was the only pollster in the group with the courage to poll the difficult-to-predict Alaska Senate race. PPP also showed less bias (error in favor of a particular party) than ANY of the other pollsters in the group.
The lowest performer, and by a not-insignificant margin, was Rasmussen. I chronicled Rasmussen's electoral woes this cycle (many of which, in my opinion, appear self-inflicted) back on Wednesday.
A couple of other pollsters were rated higher than PPP, but that lofty rating, based on an examination of the whole cycle, might have been overstated.
You see, Silver only grades pollsters on the final three weeks of the campaign. There is an operant logic here, of course: dynamics of a race often change late in the game, and rating a pollster on their early polling might be considered akin to grading a history test based on only the first 20 questions out of the 100 on the test.
Fair enough. And, most of the time, it wouldn't be an issue. Pollsters that are strong at the beginning tend to be solid throughout. Duds tend to stay duds.
But, this cycle, Nate's pollster report card, because of the constraints of counting only the final 21 days, gives credit to a pair of pollsters that deserve enormous asterisks on their plaques.
In short, calling SurveyUSA and Quinnipiac the "most accurate" pollsters of 2010 is a little laughable. Being marginally closer to the mark in late October should not absolve the two outfits for producing, without a doubt, some of the worst non-Rasmussen polling of the cycle.
Let's start with SUSA. One of the few specific beefs I have with Nate Silver's analysis of SUSA came here:
In some of the house races that it polled, SurveyUSA’s results had been more Republican-leaning than those of other pollsters. But it turned out that it had the right impression in most of those races — anticipating, for instance, that the Democratic incumbent Jim Oberstar could easily lose his race, as he eventually did.
Leaving aside one adjective that is somewhat ridiculous (a 48-46 race is not indicative of someone "easily" losing their seat), there is a bigger problem. SUSA's GOP leaning in House races led to some results that were far off the mark. Just ask "Republican Congressmen" Scott Bruun, Ilario Pantano, Andy Vidak, and David Harmer. All four were given leads by SUSA. Two of the four have already lost, and the other two are awaiting their fate via absentee ballots.
Most incredulous, however, is one race whose most egregious sins fell outside the window of Silver's interest, since they occurred during the summer.
SurveyUSA's treatment of the Virginia 5th district should absolutely disqualify them from being considered "accurate" in the 2010 cycle.
In the world of the Silver analysis, VA-05 winds up being a fairly accurate district for SUSA, since their final poll indicated an eight-point win for Republican Rob Hurt (who actually won by a hair less than 4%).
That final poll, however, belies a summer where the SurveyUSA team claims that Democratic incumbent Tom Perriello trailed by twenty-three points in July, and twenty-six points in early September.
How far off was SUSA's early polling in VA-05? Even Republican pollsters Ayers McHenry wouldn't join them on that island. When they polled the Virginia 5th during the summer, they found Rob Hurt staked to a six-point lead (49-43).
How much damage was done to Perriello's campaign by the presumption, based in no small part by SUSA's polling, that the race was far out of reach? Even the pollsters themselves seemed concerned about their treatment of the race. In mid-October, they took the unusual step of polling the race using two different sample-gathering methods. These two methods created a six-point difference. Using the method they had used throughout the 2010 cycle yielded a 17-point Hurt lead. Using the method used in the 2008 cycle yielded a smaller Hurt lead of 11 points. For their final poll, the one Nate measured, they used the latter (RDD) method. It was a tacit admission that they were uncomfortable with their own summer data in the race.
Aside from huge errors in a number of races, SUSA was hamstrung by a peculiar problem that plagued their polling throughout, and yet remained unexplained. In the America that SUSA was polling, America's young voters just LOVED them some Republicans. Even in their late polling, this was evident. With voters aged 18-34 (a slightly different parameter than national exit polling), a final week SUSA poll gave John Kasich a wider lead among 18-34 year olds (6 pts) than he garnered overall . The exit polls painted a dramatically different picture. Here, Democrat Ted Strickland did twelve points better with young voters than he did with the electorate at large.
For Quinnipiac, meanwhile, the issue was also sampling. Quinnipiac polls less frequently than SurveyUSA, PPP, and Rasmussen. The bulk of their work is confined to five states: New York, Ohio, Pennsylvania, Connecticut, and Florida. Their late polls here, as Nate's ratings attest, were pretty close to the mark.
Their earlier polling, however, was often comically bad, and needs to have a little light shed on it.
As I noted back in September, Quinnipiac routinely was releasing surveys where the partisan breakdown was more pessimistic for Democrats than even the partisan breakdowns from the 1994 exit polls. As I wrote at the time of their polling in Ohio, for example:
While Quinnipiac does not break down their samples by these characteristics the way that SUSA does, we can look at their breakdowns of the trial heat and figure a few things out. And what one can figure out quickly is that the Q poll is assuming an electorate that is considerably more Republican than any election in history. Even if you apply the 1994 voter turnout (37 D/37 R/26 I) to the partisan breakdowns of their mid-September poll, you get dramatically different numbers in the two marquee races.
Indeed, the margins, when retrofitted for the 1994 exit polls, were nearly cut in half, showing that Quinnipiac was presuming a FAR more Republican electorate than the one that showed up in 1994 (and, as it turned out, showed up in 2010).
They had the exact same problem in Connecticut and Pennsylvania. This might explain their Linda McMahon boomlet in late September (when they had the Republican down only three to Blumenthal), their Carl Paladino boomlet at around the same time (Cuomo up by just six!), and their being particularly bearish on both Sestak and Onorato earlier in the cycle.
What is jarring is that the voting behavior of the partisans in Quinnipiac's polling scarcely moved as the campaign grinded towards its conclusion. The only thing that changed, indeed, were Quinnipiac's assumptions about who would show up.
See for yourself: compare Quinnipiac's September poll in Ohio's gubernatorial race (which had John Kasich leading Democrat Tom Strickland by 17 points) with their poll on Election Eve. Notice that, on the margin, the bases moved a little bit, while Independents scarcely moved at all. Yet...somehow...the margin of the race went from a 17-point Republican lead down to a single point. The fundamentals of the race hadn't changed, just who the Q poll was asking about it.
Without a doubt, both pollsters will hail the Silver/NYT seal of approval. But they should be called to account for glaring errors earlier in the cycle that are going largely unsung. Why was SurveyUSA seeing a revolution among young voters that (apparently) did not exist? Why did Quinnipiac's surge in Democratic voters (which most pollsters showed) show up to a magnitude unparalleled by other pollsters?
These are questions that need to be answered, if these are going to be the voices the pundit class will rely on for 2012 and beyond. And these are questions that wouldn't even be asked, if we only looked at the last few polls of the cycle.