I am a huge fan of Talking Points Memo and, in particular, TPM Election Central, so the following must be taken in the spirit of admiration and bonhomie in which it is offered:
You guys don’t know how to report on polls. Seriously. You’re utterly clueless.
All it takes is a shiny new press release to prompt the TPM guys to slap up a 48-point headline announcing the latest devastating news from the front lines of the electoral process.
The problem is that these polls do not share a common methodology and are often of uncertain provenance, so they aren’t measuring the same thing (and often, I suspect, aren’t even measuring the things they intend to), yet they are paraded across the screen to the viewer as though they represent succeeding chapters of a coherent narrative.
In the recently completed PA Primary, for example, TPM reported on poll results from no fewer than 78 polls over the course of the contest.
Six of these polls were one-time wonders. Although sometimes by a recognizable media outlet, a one time poll is as useless as a Greenpeace bumper sticker on Dick Cheney’s limo. Without a couple of attempts, you can’t even tell if the polling methodology is stable. These were:
EMILY's List (Ed. Note: Seriously, WTF?)
Mason-Dixon
Suffolk
Temple
TIME
Times/Bloomberg
The following chart summarizes the remainder. Comparing to the final results of the contest (54.5% HRC, 45.4% BHO), the metrics you want to check are A) who got the spread about right, as close as possible to a 9.1 point spread (note to TPM: 9.1 is not 10, you can look that up), and you also want to see B) who had the smallest margin of ‘Undecideds’ ... we are trying to project how people will vote, not how long they can dither. The gray semicircle on the y-axis represents the Actual finish (centers on the 9.1 point spread ... Undecideds are Zero on election day).
The winners are Quinnipac, SurveyUSA, and Zogby/Newsmax. The lesson here is that whoever is covering the News Desk should consider not publishing anybody that hasn’t yet got at least a little bit of history, like this, of doing well.
The losers show a variety of problems. By 4/20, PPP had somehow determined that the race was going to Obama. In a dead heat that is an excusable mistake, but HRC won by over 9 points! Similarly, ARG’s bubble seems to be filled with helium. Muhlenberg was actually on its way to nailing it, but their last poll was on 4/2. The intern must have gone on spring break.
Mark Penn has demonstrated that numbers are more often than not a favorite weapon of flim-flam men and charlatans. Which is a shame, because basic statistics can be a great revealer of truth, but only if shared by a fourth estate that knows how to select and report on a quality product.
Cross posted from The Horse You Rode In On