This is Part III of my series, Grading the Pollster. My other two chapters on Mason-Dixon and Rasmussen are here:
http://www.dailykos.com/...
http://www.dailykos.com/...
In this diary, I take a look at Survey USA. SUSA often confounds observers with its odd internals and write-ups. It sometimes appears as if they are performing alchemy, not polling as a statistical science. They very badly need to hire a black pollster, because their internals for African Americans are terribly skewed towards the GOP, which indicates that they're not able to get reliable or large enough subgroup samples. However, SUSA has shown in the last few election cycles an uncanny ability to be more right than wrong, while still maintaining their reputation for putting some real clunkers out there.
In this election, SUSA overall did pretty well. Though they did not poll in Colorado or Nevada, two of the most important battleground state red to blue switches, in the states they did poll, they got the top line result and trends right, even if they were a little late in coming to the party in some states.
- Pennsylvania: Actual Result: Obama 55 McCain 44. SUSA final poll: Obama 52 McCain 43
SUSA was pretty close to the final result, leaving 5% undecided, and with Obama appearing to win a substantial majority of the undecided voters. The final poll missed Obama's number by 3 points and McCain's by 1, within the MOE. SUSA had strange internals for PA all year, stating that there were an inordinate number of party cross-over votes. PA did have some odd regional results, which could explain some of SUSA's internal findings. McCain outperformed Bush and Obama underperformed Kerry in the southwest corner of PA, marking one of the few battleground states where such a trend occurred. However, Obama outperformed Kerry and McCain underperformed Bush just about everywhere else in PA, from Erie in the northwest corner, to Centre County (State College) to Dauphin County (Harrisburg) to NEPA and SEPA, Obama flat out wiped the floor with McCain as his mop. Obama didn't improve substantially in Philly (though he got more total votes but not over the 500k margin that had been predicted). His improvement came in the suburbs and other major cities of the state. In this election PA may have turned from a battleground blue state to a more solidly blue state along the lines of New Jersey, as the urban and suburban counties are now more decidedly blue and they overwhelm the rural areas.
- Florida: Actual Result: Obama 51 McCain 48. Final SUSA Poll: Obama 50 McCain 47.
Though SUSA had been more skeptical of Obama's strength in Florida than others during this election cycle, they did pretty much get this one right in its final poll. Other polls (Quinnipiac, PPP) had marked the pro-Obama trend in the state long before SUSA came to the dance. Nonetheless, SUSA did pick it up at the end and that counts for a lot.
- Ohio: Actual Result: Obama 51 McCain 47. Final SUSA Poll: Obama 48 McCain 46.
SUSA's result was within the MOE but somewhat understated Obama's strength of support as many other state polls had Obama exceeding 50%. SUSA's polls showed a general tightening towards the end, as did others. However, Obama did win the state by over 200,000 votes, larger than Bush's 2004 margin of victory and representing an overall 320,000 margin swing from Kerry's 2004 finish. I think this race was much steadier than some of the polls claimed, and many polls had a hard time determining the impact of early voting. Ohio was a great example of how Obama's campaign was built to win the close ones, and this one didn't turn out to be that close as it was an early call on election night.
- Virginia: Actual Result: Obama 53 McCain 46. Final SUSA Poll: Obama 50 McCain 46.
SUSA was again within the MOE on this one, as they were on the lower end of a 4-7 point range that had been pretty consistently reflected in the polling. SUSA did seem to have a tendency to slightly understate Obama's performance, while hitting McCain's number with a higher degree of accuracy.
- North Carolina: Actual Result: Obama 50 McCain 49. Final SUSA Poll: McCain 49 Obama 48.
NC was the second closest race in the country and SUSA did accurately predict a close result. However, they got the winner wrong (even if they were in the MOE) and SUSA's famously odd internals probably cost them on this one. The final SUSA poll had McCain winning independents by 7 points, when other pollsters (like PPP) had Obama consistently ahead among indies. Furthermore, it also appears that SUSA overestimated McCain's performance in the Charlotte area, when the end result was closer to a tie.
- Indiana: Actual Result: Obama 50 McCain 49. Final SUSA Poll: Obama 49 McCain 45.
The final SUSA poll was taken about 2 weeks before election day so it was in a sense outdated by the time the votes were cast. Other polls more accurately predicted the tightness and even the actual result (PPP and Selzer being two examples). Nonetheless, SUSA deserves credit for picking up the pro-Obama trend in the state (which many thought a pipe dream). Indiana is a rare example of where SUSA underestimated McCain's final number.
- Missouri: Actual Result: McCain 50 Obama 49. Final SUSA Poll: McCain 48, Obama 48
SUSA was one of four polls that put the closest race in the country at a tie. In that sense, SUSA got this one right. Missouri finally ended its claim to being a national bellweather (this status ended in 2000, but 2008 makes it official). The MO result was most disappointing, as in a Democratic year, Missouri should have been a more solid lean blue state, but the fundie element in MO has taken over the state and has made it a lean red battleground. One hopes MO will come to its senses in 4 years as demographics change in favor of the Democrats.
- New Mexico: Actual Result: Obama 57 McCain 42. Final SUSA Poll: Obama 52 McCain 45.
SUSA got the winner right but significantly underestimated the shockingly large Obama margin of victory. PPP nailed this state and no one else did. It would appear that SUSA underestimated both Obama's share of the white vote and latino vote as Obama won Bernalillo County (Albuquerque) by an amazing 22 points. So much for the Bradley effect.
- Minnesota: Actual Result: Obama 54 McCain 44. Final SUSA POll: Obama 49 McCain 46.
SUSA always had screwy polls in Minnesota and their final poll was a real clunker. Obama did not make even one trip to MN in the general election and won the state with ease. SUSA made this race seem far more contested than either campaign believed. There is a big difference between a 3 point lead and a 10 point lead (actually, the result was 10.24 points, so the margin was closer to 11 points than it was to 9).
- Wisconsin: Actual Result Obama 56 McCain 42. Final SUSA Poll: Obama 55 McCain 39.
Unlike Minnesota, SUSA got Wisconsin pretty much right, accurately predicting a big double digit margin of victory. Winning WI by this kind of margin after Kerry barely scraped by with an 11,000 vote margin is a real sign of the strength of Obama's campaign. He was not threatened in any blue state.
- Iowa: Actual Result: Obama 54 McCain 45. Final SUSA poll: Obama 55 McCain 40.
Iowa is perhaps the only example where the margin was smaller than most polls had predicted. Obama was clearly willing to sacrifice a few points in some states to go after Indiana and North Carolina and it worked. McCain spent a lot of time in Iowa and that could account for the slightly narrow than expected margin. I believe Obama made one visit following the convention.
- New Hampshire: Actual Result: Obama 54 McCain 45 (margin closer to 10 than to 9). Final SUSA Poll: Obama 53 McCain 42.
SUSA's margin was off by about 1 point. Obama won every county in the state, a sign of true dominance. The SUSA poll suggests that more undecided voters shifted to McCain than Obama, but that appear to be nothing more than statistical noise. Obama's victory in NH was impressive (After all a black guy won the majority of the white vote in a fairly conservative, all white state).
Overall, SUSA had a pretty good election night 2008. During the year, they had some trouble spotting pro-Obama trends in some states but got things right late in the game. SUSA had a far better night than Rasmussen or Mason-Dixon, though still showing some anomalous results. I would like to see them get more consistency in regional and demographic sub group polling, but somehow their numbers tend to work oumore often than not. I give them a B- grade for being fairly accurate, but they lost some points for not polling in CO and NV, for missing some trends in FL and OH and for missing the mark somewhat in Minnesota and North Carolina.