Exit Poll Special: Febble's Fancy Function
Fri Apr 29, 2005 at 01:19:38 PM PDT
[editor's note, by DemFromCT] The data reports by Freeman, Brady, USCV, etc. can be found at the National Research Commission on Elections and Voting website. Update [2005-4-30 12:29:14 by DemFromCT]:: Febble's paper can also be found here and here.
The Mystery Pollster has a new post up today highlighting the work of a dedicated Daily Kos poster named Febble. As the MP puts it:
Regular readers of this blog may know her as
"Febble," the diarist from DailyKos. Her real name is Elizabeth Liddle, a 50-something mother and native of Scotland who originally trained as a classical musician and spent most of her life performing, composing and recording renaissance and baroque music. She also studied architecture and urban design before enrolling in a PhD program in Cognitive Psychology at the University of Nottingham where she is currently at work on a dissertation on the neural correlates of dyslexia. In her spare time, she wrote a children's
book and developed in interest in American politics while posting on the British Labor party on DailyKos. A
self proclaimed "fraudster" she came to believe our election "may well have been rigged," that "real and massive vote suppression" occurred in Ohio where the recount was "a sham" and the Secretary of State "should be jailed."
She is, in short, perhaps the least imaginable person to have developed a computational model that both suggests a way to resolve the debate about whether the exit polls present evidence of fraud and undermines the central thesis of a paper by a team of PhD fraud theorists.
But that's exactly what she's done.
Assignment editors, if you are out there (and I know you are), while it is a bit complex and geeky, this is an amazing story. Read on...
Here's a story that takes us back to the afternoon of November 2, 2004 and the leaked afternoon exit polls that showed Kerry ahead, Bush trailing. A nice summary of that, and the U.S. Count Votes report that followed was put up Monday on washingtonpost.com by Terry Neal, their senior political reporter:
The methodology and math of the study are far too complicated to get into in detail here. But here is
a link to the entire study for your reading pleasure.
Among other things, the study reports that some of the largest discrepancies between exit polls and final vote tallies occurred inexplicably in battleground states.
"This discrepancy between exit polls and the official election results has triggered a controversy which has yet to be resolved," according the report.
Last year's exit poll was conducted by Edison Media Research and Mitofsky International for ABC, NBC, CBS, CNN, Fox News and the Associated Press, and many reports, including one from Edison and Mitofsky, have found flaws in the poll.
The U.S. Count Votes analysis noted that Edison-Mitofsky's "national poll results projected a Kerry victory by 3.0 percent, whereas the official count had Bush winning by 2.5 percent. Several methods have been used to estimate the probability that the national exit poll results would be as different as they were from the national popular vote by random chance. These estimates range from 1 in 16.5 million to 1 in 1,240. No matter how one calculates it, the discrepancy cannot be attributed to chance."
Even Edison and Mitofsky have acknowledged that the problem between exit polls and the vote count could not be explained simply by statistical error. One of their primary explanations: Kerry voters were more likely than Bush voters to participate in the exit poll.
Meanwhile, on Daily Kos , some very interesting things were happening. Following the election, the exit poll discrepancy was discussed at length and in detail both on the front page and in multiple diaries, with varying points of view. Many posters showed up with regularity in the exit poll diaries out of interest and curiosity. Some, like myself, began researching past exit poll performance for clues as to what had happened. Some of the more statistically-minded were drawn to the issue. One of them, a British poster named Febble, began analyzing the available data in hopes of clarifying the exit poll section in georgia10's well-researched effort (Eye On Ohio: The Informed Citizen's Guide To The 2004 Election) from early January. One of her main objections was based on relying on Steve Freeman's probability estimates of the exit polls being wrong (way too high, in Febble's opinion, referenced in this diary).
From there, Febble began thinking about the exit poll discrepancy in some new and original ways. Aided by multiple comments and the discussion of an interested audience, Febble reviewed the previous history of exit poll probabilities (BREAKING: One in Nine Billion probability that Exit polls could have happened by chance!) showing a distinct and clear pattern of Dem bias in the past, and that `probability' is a poor function to judge exit polls by.
By this time, others including Mark Blumenthal (aka the Mystery Pollster), and Rick Brady (a grad student in city planning and a frequent poster at the Mystery Pollster site) had weighed in with their own critiques.
Febble herself had been an early participant in USCV and remained in contact with many of the math and stats people involved in the effort. Meanwhile, she began running computer simulations trying to model various changes in partisanship to see how that might affect the assumptions underlying the USCV paper. That diary in early April, New: Snark-free Exit Poll analysis, presented a novel and original way of looking at the available exit poll data and came up with what Daily Kos poster RonK dubbed "Febble's Fancy Function". Mystery Pollster today details what the math is all about; with that diary, Brady and the other posters in these parts weighed in and the subsequent peer review led to revisions which are now put together in a formal academic paper, which can be found here.
What Febble has done is shown that the oft-used variable known as Within Precinct Error (WPE) is not a very good measure of sample bias. Sample bias can be introduced by such things as Bush voters refusing to talk to pollsters. In the parlance of the trade, this is known as `differential non-response' (i.e., the non-response to pollsters is different between Bush voters and Kerry voters), but it's also been widely described as reluctant (or `shy') Bush responders, or rBr (all those terms are used in various reports and critiques). It's what Edison-Mitofsky (the exit poll company) says is the likely reason for why the exit polls were wrong, and it's primarily what USCV says is `implausible', thereby supporting their conclusion that fraud is a leading possibility to explain the exit poll discrepancy. They heavily relied on WPEs to reach that conclusion, and so did many of the subsequent Daily Kos diaries.
However, Febble's Fancy Function shows that WPE obscures diagnosis of exit poll error by introducing a lot of noise into the analysis; the Fancy Function strips the noise and leaves only sampling bias. Her computer modeling of the Fancy Function demonstrates that the E-M hypothesis (that there were shy Bush voters) remains very plausible, indeed.
In addition, assuming that Febble's work stands up to peer review, it may become an equally valid way of looking at exit polls in future, in order to detect suspicious variation in sampling bias that would deserve to be looked into further.
Febble's work does not prove there was no fraud, any more than USCV's work proves there was. The exit polls are too blunt an instrument for that, and we don't have the precinct-level data that would be required for detailed analysis (what we have is state-wide averages). Nonetheless, this is a genuine contribution to the understanding of what's happened, and as such deserves recognition. If Febble's Fancy Function is applied to precinct-level data, it is quite possible (likely, even) that lingering questions about whether it is possible for 'differential non-response' to be a plausable explanation for the exit poll discrepancy would be resolved. Only E-M has that data, so further inquiries will have to wait until after mid-May, when the American Association for Public Opinion Research will have its annual meeting. Mitofsky will be there and 2004 exit polls are on the agenda.
In addition, Febble's idea happened because of this community, the forbearance and patience of the critics, the skeptics (and the skeptics of the skeptics) that allowed the free-flowing peer review over the internet that marks Daily Kos as a special place. Febble's Fancy Function was created here, and whether or not it goes on to be published and used, you've participated in the birth of an idea. That's a very neat thing to have witnessed.
[Full disclosure and adapted from the Mystery Pollster, who has a similar disclaimer: For the last two weeks, I have had the unique opportunity to watch the development of Elizabeth Liddle's work through a running email conversation between herself, Rick Brady of StonesCryOut and Mark Blumenthal from Mystery Pollster. This post benefits greatly from their input, although as always, the views expressed here are my own - DemFromCT].
