(From the diaries - I was pleased to see how many posters here recognized an instant classic when they saw one - DemFromCT)We will all be avidly watching the exit polls on Tuesday night. Some of us will simply be avid to know what they can tell us about who won. Others will be avid to parse them for evidence of skulduggery. This is an attempt to sort out fact from fiction, and help all of us understand what is going on.
First of all: there will be more than one exit poll exercise on Tuesday, and some of the smaller independent exit polls will be specifically designed to shed light on the integrity (or otherwise) of the vote-counting process. But the big one will be the Edison-Mitofsky poll for the NEP (National Election Pool], so this diary is about that.
Purpose of the NEP exit poll
The NEP exit polls are designed primarily to answer these questions:
http://www.exit-poll.net/...
* WHO voted for each candidate
* WHY voters in your area made critical choices
* WHERE geographical differences on candidates and issues were a factor
They are also designed to allow members to "call" state results (Senate and Governor in 2006), once it is unambiguously clear (with 99.5%) confidence who is ahead.
Note that "verifying the integrity of the election" isn't one of the goals of the survey, and this is important, because, whether we wish it were otherwise or not, the exit pollsters assume that the vote count is correct.
The poll questions are addressed by means of a substantial questionnaire completed by what is designed to be a representative sample of voters. By cross-tabulating characteristics of the voters (age, race, sex, etc) with their answers to questions regarding their vote, and their reasons for their vote, a picture emerges as to who, where, voted for whom and what, and why. This is extremely interesting information.
Getting a representative sample: sampling error
But the accuracy of the information depends on how truly representative the sample is. And, unfortunately, it is remarkably difficult to get a truly representative sample of anything, let alone people. Ideally, one would put every voter into an enormous barrel, shake the barrel, and get a designated toddler to pull out a large number of voters at random, and administer the questionnaire under duress, and with the aid of a truth drug. If this was done, the only "error" in the poll would be "sampling error" - the variability you'd get if you repeated the exercise several times with several toddlers. And from this variability, the "Margin of Error" (MoE) would be computed. The MoE is the range of results you'd expect to get, say, 95%, or 99.5% of the time if your sample each time was completely random (the MoE will be wider for greater degrees of "confidence; e.g. the MoE for 99.5% confidence will be greater than the MoE for 95% confidence).
But, clearly, pollsters can't select that way. They can't even randomly sample every voter, as that would involve being, potentially, everywhere. So what they do is select a sample of precincts in such a way that each voter in a state has an equal chance of being selected, and then try to interview a similar sized sample of voters from each of the sampled precincts. A small precinct will have a lower chance of selection than a large precinct, but IF your small precinct is selected, you yourself will have a higher chance of being selected than you would if you voted in a large precinct, so the net result is that everyone has an equal chance of being selected. However, this form of sampling ("cluster sampling") means that there is less variability in the data than there would be if the sampling was truly random. Exit polls are, in effect, a large number of small polls, nested in states. Each precinct in the sample is a mini-poll, with a small sample size (and a large MoE), and only a relatively small number (in the tens, not hundreds) of precincts are sampled in each state. In order to give an estimate for the entire state, however (or for the nation, in the National precinct sample), the voters are considered as though randomly sampled (to give more statistical power), and the MoE is increased to compensate for the reduced observed variability.
Getting a representative sample: non-sampling error
Unfortunately, we can't stop there. Not all "error" in polls is "sampling error" even after we've allowed for "cluster sampling". There are many sources of "non-sampling" error, and these include factors that may systematically bias the polls. And to make it worse, unlike the sampling error that is expressed in the "MoE", this error can't be quantified in advance. Some sources of "non-sampling" error will tend to cancel out across precincts. For example "measurement error" - simple mistakes - or "coverage error" - a group of voters missed because the interviewer was taking a break - may cause error that favors one candidate or party in one precinct, but are just as likely to result in an error in the opposite direction in another. So while each precinct might have a "biased" result, the precinct "biases" should cancel out - sum to zero - over all. However, other sources of non-sampling error may not do so. If, for example, in a particular state, voters for one party tend to vote at crowded times, where voters are more likely to be "missed" (selected, but not interviewed because the interviewer is busy), then that party's voters may be systematically under-represented in the poll, across precincts. This will result in a discrepancy between poll and count that may well be statistically "significant".
Non-response bias
More serious still, is "non-response bias". Participation in the poll is entirely voluntary; selected voters are free to refuse if they do not want to participate. A problem therefore arises if there is any tendency for one group of voters to be less willing to participate than the other group (and we can never rule this out: there is no reason to think that people's attitude to political choices is unrelated to their attitude to pollsters, particularly pollsters sponsored by the news media). If this happens, then the pollsters are, in effect sampling from "a different population" from the total population of voters - they are sampling from that subset of voters who are willing to participate in the polls. And that subset may be more Democratic - or more Republican - than the total population.
There are measures the pollsters can - and do - take to compensate for non-response bias. Interviewers are asked to note the age, race and sex (by visual estimate) of all those who are selected for participation but who do not take part. Data analysts can then compare the age, race and sex ratios in the non-respondents to those in the respondent sample, and if a particular demographic group appears under-represented, they know that some form of "non-response bias" has occurred - and can re-weight the data to compensate. Indeed it is because of these kinds of data that we know a great deal about non-response bias. Unfortunately, only non-response bias by visible characteristics can be observed. The pollsters cannot know whether Republican or Democratic voters are over or under represented in their samples. And "non-response bias" may be subtle - if those who are reluctant to participate manage to avoid actually being selected - then we will tend to get what is called "selection bias".
Selection bias
Interviewers are given an "interviewing interval" that is designed to net a consistent sample size across precincts, for example 100 voters. So large precincts will be given a longer interviewing interval than small precincts. For example, interviewers may be asked to interview every 10th voter in a large precinct, while those interviewing in a small precinct may be asked to interview every second, or even every, voter. However, when the interviewing interval is large, and especially when a polling place is crowded, then strict "Nth voter" protocol may be more difficult to adhere to, and it may be easier for interviewers to find themselves unconsciously selecting the Nth +1, or Nth-1 voter if the Nth looks likely to refuse. Indeed, it may also simply make it easier for unwilling voters to evade the selection process. In the 2004 poll data, bias was significantly greater where N was large, and/or when the interviewer had to stand more than 25 feet from the polling place. Note that where selection bias occurs, response rate may actually be enhanced - if you tend to select voters who are more likely to agree to participate, then your completion rate will go up. Unfortunately, so may your bias.
In short, therefore; exit polls are surveys, and they are subject to both sampling and non-sampling error, and the sources of non-sampling error include sources of bias. In addition, and increasingly, the pollsters need to use telephone samples for absentee and early voters, and for all these samples - absentee, early, voters exiting the polls on election day - they need to make guesstimates about the likely turnout, based on past results, which may or may not be extrapolated correctly to the current election.
Compensating for non-sampling error
For these reasons, the pollsters have a number of data sources that they use to corroborate (or not) the results they get from the actual polls. One of these sources is pre-election polls - if the exit poll responses diverge greatly from pre-election polls, then they have reason to regard their exit poll responses as potentially biased. The pollsters became aware of such a divergence election day 2004 (as we know, from a leak by Wonkette), before a single result was available. Another indicator is the vote-returns themselves. Again, if the incoming vote returns indicate a systematic divergence from the exit poll response, the pollsters have reason to suspect bias in their sample. For this reason, they dynamically reweight their estimates of the final result as first the precinct, and then the county tabulations are reported, and only when they are sufficiently confident that their estimate is correct (statistically confident that is) do they recommend "calling" a state for one candidate or another. They also dynamically reweight their cross-tabulations from the same sources, to correct for what they assume is biased voter representation in their data. on edit: cross-tabulations are not reweighted dynamically; the first reweighting generally occurs two to three hours after state close of poll, and further reweighting is done thereafter as necessary.
So: if you want to know what the pollster's estimate of the results are, independently of reweighting by vote-returns, what you want are the cross-tabulations at close of poll in each state. These were provided by CNN in 2004 (not "leaked" as widely regarded - simply posted as an early stab at the numbers) and there is no reason that I know of, despite rumors to the contrary, why close-of-poll cross-tabulations won't be posted again on Tuesday. They may well change as the night wears on; this won't be because anyone is trying to "cover-up" a "leak" but simply because, rightly or wrongly, the official result is assumed to be correct, and any discrepancy between poll and count due to error in the poll.
And now to dispel a few myths:
"'Uncorrected' poll results won't be released in 2006"
Joe Lenski, in an interview with Andrew Kohut from Pew, said that "We're going to put in place systems in which no one, even at the networks, can view any of this data before 5 p.m. on Election Day." This should reduce the chance of information from incomplete samples being disseminated. An incomplete sample is highly likely to be biased, because voters do not arrive randomly. If you miss a late Republican rush, or late Democratic rush, you will get the wrong picture. Whether the networks will post close-of-poll cross-tabulations remains to be seen, but I have seen nothing to indicate that they won't.
"There is no evidence that Republicans are less likely to respond to exit polls"
There is plenty of evidence for pro-Democratic bias in exit polls. The best kind of evidence is experimental evidence, where the experimenter actually controls a variable that is randomly allocated. The random allocation ensures that it will be "orthogonal" to any other factor that might affect the phenomenon you are interested, in this case, discrepancy between poll and count. In two experiments that I know of, methodological factors were manipulated in order to try to increase response rates (one involved giving free folders; another involved experimenting with shorter questionnaires). In both cases, the one condition did result in different response rates, but also, surprisingly, to increased apparent Democratic bias. As the difference in bias cannot have been due to fraud (there would be no way any fraudsters could have known which precincts had free folders) then we know that the manipulated condition must have been causal - that methodological factors differentially affected the participation rate of Democrats versus Republicans.
"There is no evidence that Republicans were less likely to respond to pollsters in 2004"
There are strong correlations in the exit poll data between methodological factors (such as interviewing rate) and the magnitude of the precinct-level discrepancy. Where factors were present that would have made it easier for unwilling voters to avoid being polled (or for eager voters to volunteer) the greater was the observed discrepancy.
"'rBr' ['reluctant Bush responders'] is disproved by the fact that response rate in 2004 was higher in Republican states/precincts"
Because selection bias may result in high response rates, this "proof" is somewhat flawed from the outset. However, more importantly, it is when Democratic response rates differ from Republican response rates that bias will occur, whether the response rates are 15% and 20% or 60% and 80%. Overall response rates can tell us very little about bias.
"The 2004 exit polls indicate that many millions of votes were stolen"
There is absolutely no correlation between the magnitude of the precinct-level discrepancies in 2004 and change in Bush's vote share relative to 2000 (what UK commentators call "swing"). If a single factor, e.g. fraud, was responsible both for both the discrepancy, and for inflating Bush's vote, then you would expect the two to be positively correlated. In fact, the correlation is slightly, but insignificantly, negative. If fraud was responsible for the discrepancy in 2004 then either it was absolutely uniform (which is not the case normally made) or it was carefully targeted in precincts (not states) where Bush was expected do badly. Either way, the data does not support the inference that the discrepancy was due to fraud; heroic assumptions need to be made to make it even consistent with fraud on a very large, nationwide scale.
"Exit polls are uncannily accurate"
The precinct level data has shown a consistent Democratic bias over the last 5 presidential elections, and the causes of the bias have been well-researched. In 1992, the discrepancy was almost as large as in 2004. The reputation for "uncanny accuracy" probably, ironically, derives from pollsters' extreme caution about calling states unless they are very sure they are right - and they make sure they are sure by incorporating vote-returns into the estimates in all but the most slam-dunk of races. In the UK, where, for all our faults, we conduct pretty transparent elections, the exit polls are regarded as a bit of a joke (a cruel joke in 1992). Peter Snow, the BBC poll presenter on election night has as his catch phrase "it's all a bit of fun".
"Exit polls are used to monitor election integrity around the world"
Not as far as I know. In Ukraine, there was direct evidence of blatant fraud (acid in ballot boxes; candidate poisoned with disfiguring, potentially lethal poison). Sure, fraud will tend to play havoc with exit polls, but given that exit polls can play havoc with themselves, they can never be a primary instrument for monitoring election integrity. Indeed, here are some cautions:
http://www.cartercenter.org/...
http://www.cartercenter.org/...
http://www.cartercenter.org/...
http://jimmycarter.org/...
Take home message:
The early exit polls will give you a reasonable idea of who is winning on election night, but there is no point in expecting the results to be within any calculated "Margin of Error", as MoE calculations assume random sampling error only, and do not reflect non-sampling error, which polls inevitably contain. Therefore, even if the final results diverge "significantly" from the early poll, it won't necessarily mean there is fraud. Exit polls have too many potential sources of bias for bias ever to be ruled out. If you want to find fraud, don't try and find it in the NEP exit polls. Independent polls designed for the purpose may tell you more, but they are unlikely to be much more immune from bias than the NEP exit polls, and may be more vulnerable.
The Edison-Mitofsky FAQ is here:
http://www.exit-poll.net/...