I read the study on the exit poll failure and have a few comments if anyone might please chime in.
http://www.exit-poll.net/election-night/EvaluationJan192005.pdf
The gist of the report is that the discrepancy between the exit polls, showing a healthy lead for Kerry on election night and the actual results of a narrow Kerry loss was caused by "differential non-response" (DNR). That is, presumptively, Bush voters were more likely than Kerry voters to refuse to be interviewed by an exit pollster.
However, as I read it, this is a very unsatisfying conclusion, as there is no direct evidence presented in the report that DNR leaned one way or the other. Rather, when the measured discrepancy between the Kerry poll and actual margins, called "within precinct error" (WPE) could not be explained by any other means, DNR was simply the default explanation.
WPE in the report is positive for error favoring Bush and negative for error favoring Kerry. E.g., if the poll said Kerry led by 5% but the actual results have Kerry losing by 2%, the WPE is -7. There are lots of tables in the report trying to show correlation between WPE and selected characteristics of the interviewer, polling location and other factors.
When discussion turns towards what might have caused the DNR favoring Kerry, general handwaving ensues. One idea spreading around is that somehow young 20-something interviewers with clipboards just attract Democratic voters--differentially of course.
But even on the basis of the numbers given in the report, this conclusion seems weak:
So were Kerry voters also attracted to 55-64 year old interviewers?
I would like to see a measure of the statistical significance of the conclusion that "Older interviewers had a lower WPE than the youngest interviewers."
Moreover, since the whole idea is predicated on a supposed tendency of Bush supports to refuse to be interviewed, the following result in the study would seem to run counter: "There was no significant difference between the completion rates and precinct partisanship."
So if the premise is that Bush voters were more likely to be refuseniks, wouldn't we expect to see an effect based on precinct partisanship? And since such an effect is not observed, might not the whole premise be rejected?
Here's the data presented in the report on this score:
So where is the beef behind this DNR hypothesis, or is it simply what it appears: just the default hypothesis when no other explanation can be found?
There are so many variables of course that are hidden in such simple charts and tables. For example, maybe the age of the interviewer just happened to be clustered differently among Bush and Kerry precincts. Here's an example of just one such faulty conclusion I think in the report (where other variables are not considered). The report states:
"The WPE was greater when there were three or more precincts at the polling place."
This was attributed to confusion among precincts (where the interviewers were supposed to sample only one precinct). But Cuyahoga County data with which I'm familiar shows that the multiple-precinct polling places tended to be in the exurbs that were heavily Bush. So rather than showing an effect of number of precincts on the WPE, they might just be seeing a restatement of the main effect of greater WPE in Bush precincts.
Here are a couple of maps illustrating the distribution of multiple-precinct polling places in Cuyahoga County:
So to my mind we are still left with the basic, unexplained result, that for some reason the exit polls in heavily-Bush precincts had large errors. Here's that data from the report:
The report prefers to throw out the heavy Kerry and heavy Bush precincts since in an 80% Kerry precinct there is little room for overstating Kerry support, and vice versa. So it concentrates on the middle three groups which show a more or less consistent error towards Kerry of 6-8%.
I keep getting back to the finding that they observed no effect based on precinct partisanship. To give an average WPE of -7%, Bush voters would have to refuse to be interviewed at approximately double that or 14%, compared to Kerry voters.
That should show up in the completion/refusal graph above, but doesn't.
Seems to me that only by ignoring Occam's Razor, and multiplying causes willy-nilly, might the DNR hypothesis be maintained. For example, in an 80% Bush precinct maybe the Bushies refuse at a high rate, but that is made up for by 20% Democrats, who, outnumbered in such precincts, are more anxious than usual to stand up and be counted. So the completion rate stays constant? Suuure. As the magician says: "Notice at no time do my fingers leave my hands."