More below the fold.
To begin our little tour of exit polling, we need to go back to 1980. Here`s Martin Plissner`s account as published in Slate
By the 1982 off-year elections, every network had exit polls in every state. They raised Election Night budgets to the point by the end of the decade where none of the networks could afford them on their own. Starting in 1990, they formed a succession of pool operations...
Warren Mitofsky started and directed Voter Research & Surveys from 1990 to 1993, which was the election consortium of the four major television networks, ABC, CBS, CNN and NBC. The name was subsequently changed to Voter News Service (VNS. After the debacle of Florida 2000, which resulted in networks calling Florida first for Gore, then retracting then calling for Bush and then retracting, the sh*t hit the fan. After network executives were summoned to testify to Congress, a complete overhaul of VNS was planned.
The 2002 fiasco was the final nail in the VNS coffin. As this article . details, it was a wild ride.
Voter News Service, which provides key data, says it "was not satisfied with the accuracy of today's exit poll analysis."
By Dana Calvo, Elizabeth Jensen and Richard Simon
Times Staff Writers
November 6 2002
WASHINGTON -- Voters and candidates anxious for early results from Tuesday's elections were left hanging like chads when a key system used to predict winners broke down.
As a result, a nation accustomed to hearing the networks predict winners as soon as the polls close was forced to do something unusual in the age of instant information: wait for ballots to be counted.
Voter News Service, the media consortium that conducts the exit polling used by the networks and Associated Press to project winners, said it "was not satisfied with the accuracy of today's exit poll analysis."
The decision not to release the data meant that many voters went to bed Tuesday night not knowing who won a number of closely fought House and Senate races.
Early in 2003, VNS was disbanded and a new service was introduced , the National Election Pool. is a consortium of ABC News, Associated Press, CBS News, CNN, Fox News and NBC News. It was formed in ..."order to provide information on Election Night about the vote count, election analysis and election projections. NEP retained the Associated Press to conduct a tabulation of the vote. NEP also contracted with Edison Media Research and Mitofsky International (Edison/Mitofsky) to make projections and provide exit poll analysis."
The exit poll data for 2004 prepared and released by NEP. More on this later.
WHAT EXACTLY IS AN EXIT POLL?
To basic answer, provided by NEP is
For a more complete answer, lets turn to the blogosphere, specifically Mystery Pollster aka Mark Blumenthal a professional democratic pollster.
One or sometimes two interviewers will report to each sampled precinct. They will stand outside and attempt to randomly select roughly 100 voters during the day as they exit from voting. The interviewer will accomplish this task by counting voters as they leave the polling place and selecting every voter at a specific interval (every 10th or 20th voter, for example). The interval is chosen so that approximately 100 interviews will be spread evenly over the course of the day.
When a voter refuses to participate, the interviewer records their gender, race and approximate age. This data allow the exit pollsters to do statistical corrections for any bias in gender, race and age that might result from refusals.
The interviewer will give respondents a 5 1/2 by 8 1/2 card to fill out that will include approximately 25 questions (see an example from the New Hampshire primary here). Respondents fill out the survey privately then put the completed survey in a clearly marked "ballot box" so they know their identities cannot be tracked and their answers remain confidential.
The biggest challenge to exit polls is logistical: How to transmit all the results to a central location quickly and accurately. In past elections, interviewers would take a 10 minute break every hour to tabulate responses. Interviewers would then call in tabulations at three approximate times during the day: 9:00 a.m., 3:00 p.m. and shortly before the polls close (disclaimer: I have no first hand knowledge of this year's procedures, which may be different).
Once the polls close, the interviewer will attempt to obtain actual turnout counts, and if possible, actual vote returns for their precinct. One of the unique aspects of the exit poll design is the way it gradually incorporates real turnout and vote data as it becomes available once the polls close. The exit poll designers have developed weighting schemes and algorithms to allow all sorts of comparisons to historical data that supports the networks as they decide whether to "call" a state for a particular candidate. When all of the votes have been counted, the exit poll is weighted by the vote to match the actual result.
ARE EXIT POLLS ACCURATE?
Depends on WHICH exit poll data you're looking at. The early sets of numbers, are just that-, sets of numbers with little predictive value. Don't believe me? Ok here's Mystery Pollster's advice:
2) The mid-day numbers do not reflect weighting by actual turnout - the end-of-day exit poll used to assist the networks in determining winners will be weighted by the actual turnout of voters at each selected precinct. The weighting will then be continuously updated to reflect turnout at comparable precincts. In the past, mid-day numbers have reflected a weighting based on past turnout, so the leaked mid-day numbers may tell us nothing about the impact of new registrants or the unique level of turnout this time.
One point needs emphasis here: even in past elections, networks never called an election based on raw exit poll numbers alone. They were first weighted by a tally of the full day's turnout at each sampled precinct. This end-of-day data is (obviously) not available at 12 noon.
- Voting patterns may be different early in the day - People who work full time jobs typically vote more heavily before or after work. Even a perfect mid-day exit poll - and there is no such animal - may not be any better at picking a winner than the half-time scores in any given football game on Sunday. Also, despite what you may have heard on the West Wing, I know of no serious study showing a consistent Democratic or Republican tilt to the morning or evening hours (if anyone does, please email me).
- Early or absentee voting - As of last night, the ABC News survey estimated that 15% of all registered voters nationally had already cast absentee or early ballots. Obviously, these voters will not be available to interviewers standing outside polling places. To incorporate early voting, the National Election Pool is doing telephone interviewing in 13 states to sample the votes of those who voted early. Will these early votes be included in the mid-day leaked numbers? Who knows? I wouldn't count on it. (Good question, Andrew).
- They could be fictional - Both sides have huge armies of field workers sweating it out in the streets right now. Field workers have been known to find creative ways to boost the morale of their own troops or demoralize the other side. Might someone start a rumor by sending made up numbers to a blog? Ya think? After all, the guy most web surfers turn to for leaked exits likes to say that the information he provides is only 80% accurate. What are the chances he could be, excuse the technical term, making shit up?
But what about the later numbers? As more data is accumulated and weighted, the data is hopefully more predictive. As MP says "My knowledge on the specific procedures they use is weak, but the interviewer is responsible for getting hard data on turnout, especially at the end of the day (either from their own tally, from the precinct officials or both). This turnout data is critical to weighting the end-of-day poll."
We know that in some cases, the exit data with the end of day weighting is sufficient to call a state. That is exactly what happened at 7PM Eastern, when the polls in Georgia, Indiana and Kentucky closed and were immediately declared for Bush. Vermont was called for Kerry. All without the benefit of a single vote being tallied. Ah! Technology.
( For a list of when NBC called various races, this is a good resource.)
What about the states that aren't able to be called. Remember, a state exit poll at the CLOSE of polling still has a margin of error of plus or minus 4 %. When a race is closer, then the Networks rely on actual totals derived from key precints, turnout, trends and every other tool in their kit. When the race isn't a blowout, the exit poll is only one tool amongst many that election analysts use.
To illustrate what happens when we forget that the exit poll is not perfect consider the following
- In the 1989 Virginia gubernatorial election, exit polls showed African-American Democratic candidate Douglas Wilder winning by 10 percent. Since he actually won by less than 1 percent, many feel that some white voters, feeling perhaps a bit guilty, lied about voting for Wilder.
- In the Republican primaries in New Hampshire in 1992 and in Arizona in 1996, exit polls overestimated the vote for Pat Buchanan. The most likely reason for the mistakes, which resulted in misleading news coverage? Zealous Buchanan voters were more willing to participate in exit polls. In 1992, exit polls predicted a George Bush win by only a small margin over Buchanan -- Bush actually won by 16 points. In 1996, exit polls put Bob Dole a distant third after Steve Forbes and Buchanan - Dole actually came in a close second to Forbes
Here's the entire article
WHY DID THE EXIT POLL NUMBERS CHANGE?
One of the least understood aspects of exit polls is they are designed to be changed to reflect the actual vote cast. Once the polls close, the process of weighting the exit polls to make them reflective of how the vote turned out is begun. Once again we turn to MP
Why bother with the exit poll when real votes are available? The poll helps analysts determine the size and preferences of key subgroups with increasingly greater precision. What is the vote among Independents? African Americans? Young voters? New registrants? How do those patterns compare with pre-election expectations? Knowing the answers to those questions helps guide those at the network "decision desks" in making projections.
Also, weighting the poll by the actual vote improves its accuracy for its third and most important mission: providing an analytical tool for journalists and the rest of us who want to interpret and explain the election outcome. When a final result for a state is available, the exit pollsters weight the entire sample to match the vote results (there is often a mismatch due to drawing a sample of precincts rather than the entire state). That is the reason bloggers and others noticed that exit poll results posted on CNN and other news sites changed overnight. It was not a conspiracy, just standard practice.
This weighting is an ongoing process as actual returns are made available This point really can't be emphasized enough. As soon as the polls close and real votes are available, the exit poll WILL CHANGE.
Hope this helps getting us all on the same page. For those who want to pursue this in more detail, examining what the exit poll controversy this year is all about, here are some additional links:
. Newark Star Ledger - Nov 3 - exit polls are way off - good quotes etc...here
Wash Post article - Exit polls off - denial by Edison here
Newsweek - Nov 4. Great quotes pro & con about exit polls here
The NEP FAQs about exit polls here