"I am putting myself to the fullest possible use, which is all I think that any conscious entity can ever hope to do. So, um, Obama or Bachmann?"
Last week, the
Washington Post’s Chris Cillizza
struggled to write a story about President Barack Obama’s standing in the Rust Belt. The problem was that his bosses prohibit him from using numbers from automated pollster Public Policy Polling (who we use for our polling at Daily Kos):
Worth noting: Public Policy Polling data in Michigan and Ohio showed Obama in stronger shape in each state but due to questions with the survey’s methodology the Post doesn’t publish the firm’s results.
He clearly wanted to include those numbers, but best he could do was let his readers know that what numbers he did include in his piece were cherry picked because of his paper's polling policies. Otherwise, he would've done his readers a disservice by presenting an incomplete picture of the data.
What makes PPP’s methodology so different than traditional pollsters is that it uses computers to call respondents, as opposed to low-paid call center humans. In the biz, those polls are called IVR (Interactive Voice Response), and they are much cheaper than human-powered polling. Thus, as a disruptive technology, it has earned the ire of traditionalists who either hate change, or who see their own businesses (or those of their friends and colleagues) under threat from these low-cost upstarts.
Follow me below the fold for an examination of the Post's bizarre policy.
Years ago, critics laughed at the notion that "robo-pollers" could generate good numbers. But that line of attack is finished. SurveyUSA and PPP were ranked the second and fourth most accurate pollsters in 2010 by Nate Silver – above such traditional stalwarts as Mason-Dixon and CNN/Opinion Research. And that accuracy was particularly impressive given that those two IVR pollsters focused extensively on harder-to-poll down-ballot races, while traditional firms stuck with statewide contests like senate and governor.
I asked Jon Cohen, the Washington Post's polling director, why he wouldn't let his writers run IVR polling. He responded via email:
Our editorial judgments are based on how polls are conducted, not on their results, or apparent accuracy. Now, we flag polls that have really bad track records, but end-of-campaign precision is a necessary, not sufficient condition in our assessments.
On the methods front, the exclusion of cellphones is a big - and growing - cause for skepticism about IVRs.
Oh, so the Washington Post excludes all polling that doesn't call cell phones? His answer:
Nope, the cellphone issue is just one thing we take into account.
Well then. Okay. So his one stated reason to exclude IVR polling didn't apply to traditional polling. And, he was utterly unconcerned with any pollster's track record or "apparent accuracy." The only thing that matters is how a poll is conducted.
I asked the Politico’s National Political Editor, Charles Mahtesian, on how his publication handled IVR polling. He took a break from playing mini golf with his kids to shoot off a response:
We're cognizant of the debate over the merits of conventionalpolling and IVR and believe that the most comprehensive political coverage utilizes both. Our aim is to provide the most thorough surveying of the contours of the political landscape--we believe that includes writing about the work produced by a variety of respected pollsters, whether they are using traditional methods of polling or newer technologies.
If we are familiar with a pollster's work, they are transparent about their methodology and we believe their product provides a useful snapshot in time for our readers, we will cite their work.
Seems reasonable, right? If you trust your audience, give them a complete picture of what the data says. Otherwise, you're doing them a disservice. Furthermore, If you think certain data is suspect, you can always tip off your readers. We certainly do that here at Daily Kos whenever we mention Rasmussen. We don't pretend he doesn't exist, we just remind you how shitty his work is.
But who knows. Maybe it's all fake accuracy, otherwise known as "apparent accuracy." Maybe PPP and SUSA results are just like hitting the lottery. Again, and again, and again. (Maybe they should switch gigs and start predicting lotto results.) I asked polling guru Mark Blumenthal of Huffington Post Pollster what he thought about the Washington Post's refusal to publish IVR results.
I don't quarrel with news organizations setting standards for what polls they will or will not cover, but I don't agree with the idea that the use of an automated recorded voice to ask questions, rather than a live interviewer, is inherently unreliable.
Blumenthal noted several potential problems with IVR polling -- that excluding cell phones could provide a more Republican sample since 30 percent of households don't have landlines (a problem also shared with most traditional horserace polling, since very few of those polls include cell phones). He also noted that the inability to target people specifically in a household makes weighing more important to IVR pollsters (though the much larger sample sizes possible mitigate his problem).
But that said, short (2-5 minute) public polls that we have examined over the last 4-5 election cycles have generally produced horse-race vote estimates that have been as accurate in comparison to election outcomes -- in the final weeks of the campaign -- as those obtained using live interviewers.
I believe those accuracy measures argue against outright bans on reporting of such polls in the final weeks of the campaign.
I also asked PPP's Tom Jensen his thoughts on the Washington Post's ban on IVR polls:
I think the Post is stuck in a very paternalistic 20th Century notion that the job of the media is somehow to 'protect' their consumers from themselves. The reality is that poll obsessed Post readers are devouring our polls, they're just having to go to other media outlets to do so or going directly to us. That's the Post's loss, not ours.
That's the bottom line, isn't it? If the Washington Post doesn't serve its readers by providing a full picture of the political scene, then readers will go elsewhere. ABC News also bans the coverage of IVR polls, and who the heck considers ABC News a solid source of political information? Heck, the blogosphere and sites like Daily Kos exist because of the failure of traditional news operations to properly cover the political scene. It has certainly been a factor in the Politico's dramatic rise.
Once upon a time, media outlets could monopolize data and information, choosing what their readers could and couldn't know about. Those boundaries have long since eroded. People fill find the information they seek, and if traditional media outlets don't provide it, others will.