Skip to main content

   There seems an inordinate amount of confusion here at dKos in claims and counter-claims of what the Exit polls show. This diary is my attempt to give an overview of exit polling. What it can be used for, what it shouldn't be used for and  what it really really means. By way of disclaimer I am not associated with polling or statistics in any way shape or form  Just an ordinary Kossack who wanted to understand a topic a little better
More below the fold.

A LITTLE HISTORY

To begin our little tour of exit polling, we need to go back to 1980. Here`s Martin Plissner`s account as published in Slate

Back in the 1960s, when the networks first began projecting winners state by state, they had to wait for real votes, even in safe states, before they made a call. In 1980, however, NBC, without any notice to the others, changed the rules. They used exit polls--scientifically sampled surveys of voters leaving their polling stations throughout Election Day--to make a bunch of calls in states where the polls closed at 8 p.m., before any votes were counted. This method enabled NBC a few minutes later to post a triumphant "Reagan elected" graphic which no competitor could match for hours. Jobs at the other networks were suddenly in jeopardy.
By the 1982 off-year elections, every network had exit polls in every state. They raised Election Night budgets to the point by the end of the decade where none of the networks could afford them on their own. Starting in 1990, they formed a succession of pool operations...

   Warren Mitofsky started and directed Voter Research & Surveys from 1990 to 1993, which was the election consortium of the four major television networks, ABC, CBS, CNN and NBC. The name was subsequently changed to  Voter News Service (VNS. After the debacle of Florida 2000, which resulted in networks calling Florida first for Gore, then retracting then calling for Bush and then retracting, the sh*t hit the fan. After network executives were summoned to testify to Congress, a complete overhaul of VNS was planned.
   The 2002 fiasco was the final nail in the VNS coffin. As this article . details, it was a wild ride.
Breakdown Leaves Networks Hanging
Voter News Service, which provides key data, says it "was not satisfied with the accuracy of today's exit poll analysis."
By Dana Calvo, Elizabeth Jensen and Richard Simon
Times Staff Writers
November 6 2002
WASHINGTON -- Voters and candidates anxious for early results from Tuesday's elections were left hanging like chads when a key system used to predict winners broke down.
As a result, a nation accustomed to hearing the networks predict winners as soon as the polls close was forced to do something unusual in the age of instant information: wait for ballots to be counted.
Voter News Service, the media consortium that conducts the exit polling used by the networks and Associated Press to project winners, said it "was not satisfied with the accuracy of today's exit poll analysis."
The decision not to release the data meant that many voters went to bed Tuesday night not knowing who won a number of closely fought House and Senate races.

  Early in 2003, VNS was disbanded and a new service was introduced , the National Election Pool. is a consortium of ABC News, Associated Press, CBS News, CNN, Fox News and NBC News. It was formed in ..."order to provide information on Election Night about the vote count, election analysis and election projections. NEP retained the Associated Press to conduct a tabulation of the vote. NEP also contracted with Edison Media Research and Mitofsky International (Edison/Mitofsky) to make projections and provide exit poll analysis."
   The exit poll data  for 2004 prepared and released by NEP. More on this later.

WHAT EXACTLY IS AN EXIT POLL?

   To basic answer, provided by NEP is

Exit polls are interviews with voters after they have cast their votes at their polling places. The polling places are a scientifically selected sample of polling places that collectively represent a state, or for the national exit poll they represent the nation. An interviewer gives every nth voter exiting the polling place a questionnaire to complete. There are questions about demographic such as gender, age, race, and issues related to the person' s vote and questions about the person' s vote choice in the different contests. Participation is voluntary and anonymous. Interviewing starts when the polls open in the morning, continues throughout the day until about an hour before they close at night.

   For a more complete answer, lets turn to the blogosphere, specifically  Mystery Pollster aka Mark Blumenthal a professional democratic pollster.
A quick summary of how exit polls work: The exit pollster begins by drawing a random sampling of precincts within a state, selected so that the odds of any precinct being selected are proportionate to the number that typically vote in that precinct. The National Election Pool Exit Poll, which is conducting the exit polling for the six major networks today, will send exit pollsters to 1,495 precincts across the country.
One or sometimes two interviewers will report to each sampled precinct. They will stand outside and attempt to randomly select roughly 100 voters during the day as they exit from voting. The interviewer will accomplish this task by counting voters as they leave the polling place and selecting every voter at a specific interval (every 10th or 20th voter, for example). The interval is chosen so that approximately 100 interviews will be spread evenly over the course of the day.
When a voter refuses to participate, the interviewer records their gender, race and approximate age. This data allow the exit pollsters to do statistical corrections for any bias in gender, race and age that might result from refusals.
The interviewer will give respondents a 5 1/2 by 8 1/2 card to fill out that will include approximately 25 questions (see an example from the New Hampshire primary here). Respondents fill out the survey privately then put the completed survey in a clearly marked "ballot box" so they know their identities cannot be tracked and their answers remain confidential.
The biggest challenge to exit polls is logistical: How to transmit all the results to a central location quickly and accurately. In past elections, interviewers would take a 10 minute break every hour to tabulate responses. Interviewers would then call in tabulations at three approximate times during the day: 9:00 a.m., 3:00 p.m. and shortly before the polls close (disclaimer: I have no first hand knowledge of this year's procedures, which may be different).
Once the polls close, the interviewer will attempt to obtain actual turnout counts, and if possible, actual vote returns for their precinct. One of the unique aspects of the exit poll design is the way it gradually incorporates real turnout and vote data as it becomes available once the polls close. The exit poll designers have developed weighting schemes and algorithms to allow all sorts of comparisons to historical data that supports the networks as they decide whether to "call" a state for a particular candidate. When all of the votes have been counted, the exit poll is weighted by the vote to match the actual result.
This is just a small flavor of the wealth of information that he provides. Go read here.

ARE EXIT POLLS ACCURATE?

   Depends on WHICH exit poll data you're looking at. The early sets of numbers, are just that-, sets of numbers with little predictive value. Don't believe me? Ok here's Mystery Pollster's advice:

1) It is still just a survey - Even when complete, an exit poll still has the same random variation as any other survey. NEP says typical state exit polls will have a sampling error when complete of +- 4% at a 95% confidence level, and +- 3% for the national exit poll. Even if comparable to the final numbers - which they are decidedly not - the mid-day leaked numbers would have much greater error, perhaps +/- 7% or more.
2) The mid-day numbers do not reflect weighting by actual turnout - the end-of-day exit poll used to assist the networks in determining winners will be weighted by the actual turnout of voters at each selected precinct. The weighting will then be continuously updated to reflect turnout at comparable precincts. In the past, mid-day numbers have reflected a weighting based on past turnout, so the leaked mid-day numbers may tell us nothing about the impact of new registrants or the unique level of turnout this time.
One point needs emphasis here: even in past elections, networks never called an election based on raw exit poll numbers alone. They were first weighted by a tally of the full day's turnout at each sampled precinct. This end-of-day data is (obviously) not available at 12 noon.
  1. Voting patterns may be different early in the day - People who work full time jobs typically vote more heavily before or after work. Even a perfect mid-day exit poll - and there is no such animal - may not be any better at picking a winner than the half-time scores in any given football game on Sunday. Also, despite what you may have heard on the West Wing, I know of no serious study showing a consistent Democratic or Republican tilt to the morning or evening hours (if anyone does, please email me).
  2. Early or absentee voting - As of last night, the ABC News survey estimated that 15% of all registered voters nationally had already cast absentee or early ballots. Obviously, these voters will not be available to interviewers standing outside polling places. To incorporate early voting, the National Election Pool is doing telephone interviewing in 13 states to sample the votes of those who voted early. Will these early votes be included in the mid-day leaked numbers? Who knows? I wouldn't count on it. (Good question, Andrew).
  3. They could be fictional - Both sides have huge armies of field workers sweating it out in the streets right now. Field workers have been known to find creative ways to boost the morale of their own troops or demoralize the other side. Might someone start a rumor by sending made up numbers to a blog? Ya think? After all, the guy most web surfers turn to for leaked exits likes to say that the information he provides is only 80% accurate. What are the chances he could be, excuse the technical term, making shit up?
.

   But what about the later numbers?  As more data is accumulated and weighted, the data is hopefully more predictive.  As MP says "My knowledge on the specific procedures they use is weak, but the interviewer is responsible for getting hard data on turnout, especially at the end of the day (either from their own tally, from the precinct officials or both). This turnout data is critical to weighting the end-of-day poll."

   We know that in some cases, the exit data with the end of day weighting is sufficient to call a state. That is exactly what happened at 7PM Eastern, when the polls in Georgia, Indiana and Kentucky closed and were immediately declared for Bush. Vermont was called for Kerry. All without the benefit of a single vote being tallied. Ah! Technology.
( For a list of when NBC called various races, this is a good resource.)
.
   What about the states that aren't able to be called. Remember, a state exit poll at the CLOSE of polling still has a margin of error of plus or minus 4 %. When a race is closer, then the Networks rely on actual totals derived from key precints, turnout, trends and every other tool in their kit. When the race isn't a blowout, the exit poll is only one tool amongst many that election analysts use.
 To illustrate what happens when we forget that the exit poll is not perfect consider the following

* VNS overstated the final Democratic vote in the 1992 presidential race. Many believe that this was due to Perot voters being relatively suspicious and unwilling to cooperate with the exit polls.
  • In the 1989 Virginia gubernatorial election, exit polls showed African-American Democratic candidate Douglas Wilder winning by 10 percent. Since he actually won by less than 1 percent, many feel that some white voters, feeling perhaps a bit guilty, lied about voting for Wilder.
  • In the Republican primaries in New Hampshire in 1992 and in Arizona in 1996, exit polls overestimated the vote for Pat Buchanan. The most likely reason for the mistakes, which resulted in misleading news coverage? Zealous Buchanan voters were more willing to participate in exit polls. In 1992, exit polls predicted a George Bush win by only a small margin over Buchanan -- Bush actually won by 16 points. In 1996, exit polls put Bob Dole a distant third after Steve Forbes and Buchanan - Dole actually came in a close second to Forbes

 Here's the entire article

WHY DID THE EXIT POLL NUMBERS CHANGE?

   One of the least understood aspects of exit polls is they are designed to be changed to reflect the actual vote cast. Once the polls close, the process of weighting the exit polls to make them reflective of how the vote turned out is begun.  Once again we turn to MP

As returns start to come in, the exit pollsters weight each individual precinct sample by the actual vote cast by all voters at that precinct. Thus, as the night wears on, the accuracy of the exit poll gradually improves.
Why bother with the exit poll when real votes are available? The poll helps analysts determine the size and preferences of key subgroups with increasingly greater precision. What is the vote among Independents? African Americans? Young voters? New registrants? How do those patterns compare with pre-election expectations? Knowing the answers to those questions helps guide those at the network "decision desks" in making projections.
Also, weighting the poll by the actual vote improves its accuracy for its third and most important mission: providing an analytical tool for journalists and the rest of us who want to interpret and explain the election outcome. When a final result for a state is available, the exit pollsters weight the entire sample to match the vote results (there is often a mismatch due to drawing a sample of precincts rather than the entire state). That is the reason bloggers and others noticed that exit poll results posted on CNN and other news sites changed overnight. It was not a conspiracy, just standard practice.

This weighting is an ongoing process as actual returns are made available This point really can't be emphasized enough. As soon as the polls close and real votes are available, the exit poll WILL CHANGE.

   Hope this helps getting us all on the same page. For those who want to pursue this in more detail, examining what the exit poll controversy this year is all about, here are some additional links:

  . Newark Star Ledger - Nov 3 - exit polls are way off - good quotes etc...here

   Wash Post article - Exit polls off - denial by Edison here  

    Newsweek - Nov 4. Great quotes pro & con about exit polls here

The NEP FAQs about exit polls here

Originally posted to recentdemocrat on Sat Nov 13, 2004 at 07:55 PM PST.

EMAIL TO A FRIEND X
Your Email has been sent.
You must add at least one tag to this diary before publishing it.

Add keywords that describe this diary. Separate multiple keywords with commas.
Tagging tips - Search For Tags - Browse For Tags

?

More Tagging tips:

A tag is a way to search for this diary. If someone is searching for "Barack Obama," is this a diary they'd be trying to find?

Use a person's full name, without any title. Senator Obama may become President Obama, and Michelle Obama might run for office.

If your diary covers an election or elected official, use election tags, which are generally the state abbreviation followed by the office. CA-01 is the first district House seat. CA-Sen covers both senate races. NY-GOV covers the New York governor's race.

Tags do not compound: that is, "education reform" is a completely different tag from "education". A tag like "reform" alone is probably not meaningful.

Consider if one or more of these tags fits your diary: Civil Rights, Community, Congress, Culture, Economy, Education, Elections, Energy, Environment, Health Care, International, Labor, Law, Media, Meta, National Security, Science, Transportation, or White House. If your diary is specific to a state, consider adding the state (California, Texas, etc). Keep in mind, though, that there are many wonderful and important diaries that don't fit in any of these tags. Don't worry if yours doesn't.

You can add a private note to this diary when hotlisting it:
Are you sure you want to remove this diary from your hotlist?
Are you sure you want to remove your recommendation? You can only recommend a diary once, so you will not be able to re-recommend it afterwards.
Rescue this diary, and add a note:
Are you sure you want to remove this diary from Rescue?
Choose where to republish this diary. The diary will be added to the queue for that group. Publish it from the queue to make it appear.

You must be a member of a group to use this feature.

Add a quick update to your diary without changing the diary itself:
Are you sure you want to remove this diary?
(The diary will be removed from the site and returned to your drafts for further editing.)
(The diary will be removed.)
Are you sure you want to save these changes to the published diary?

Comment Preferences

  •  Exit polls (none)
    Thanks, that's helpful.  One question that I haven't seen addressed:  why did Kerry's internal polling (and from what I recall, Bush's) line up with the exit polls, not the results?  Were there problems with them as well?

    Just curious.

    •  Unfortunately, neither campaign is talking (none)
      about what their internal polls are showing. Remember, in a close race 3 % margin of error means Bush really has a spread of 6 points as does Kerry. And again, I don't know what the national numbers were at the point the polls closed. Anything after that is useless, so I really don't know what the exit polls were showing except it was favorable to Kerry.

      Moderation, the noblest gift of heaven. - Euripedes

      by recentdemocrat on Sat Nov 13, 2004 at 08:09:06 PM PST

      [ Parent ]

  •  Well said, sir. It answers a lot of questions (none)
    •  Thanks, trying to get us all on the same page (none)
      Lots of misinformation flying hot and heavy regarding "the exit polls". Just trying to get my arms around it for my own piece of mind.

      Moderation, the noblest gift of heaven. - Euripedes

      by recentdemocrat on Sat Nov 13, 2004 at 08:10:59 PM PST

      [ Parent ]

      •  I, for one, could never figure out (none)
        how an exit poll changed when the real tallies came in, and how they affected each other.  Thanks for that explanation.  I'm not sure the tinfoil hat crowd will be satisfied with it, but somehow I think they are just dissatisfied, period.
  •  You seem (none)
    You seem to stress the inaccuracy of exit polls.

    They were used from 1980 forward with little or no problems.  In fact, exit polls are used in many places to check that the actual vote has not been hacked.

    The problems started happening in 1992, as you mention, with the Bush campaign.  I wasn't familiar with the problems in the republican primary.  But I am familiar with the fact that the exit polls were pretty darn accurate in most every state.  Most every state, that is, except Florida.  This from Wikipedia:

    The Voter News Service's reputation was badly tarnished by its treatment of Florida's presidential vote in the year 2000. Calling the state as a win for Gore 12 minutes before polls closed in Florida's central time zone may have affected the vote results, and inconsistent polling results caused the VNS to change its call from Gore to Bush to 'too close to call'. An attempt by VNS to use computer tallying during the 2002 congressional election was a failure, and the VNS disbanded.

    Florida 2000 was a key event.  This is exactly when exit polling--which was very trustworthy up to that point--started to go haywire.  Without understanding that, you can't fully understand why some people are very suspicious about paperless voting machines and calling for fully counting the vote.

    Just my two cents.

    •  Oops (none)
      The problems started happening in 1992...

      That should read 2000.  Got my election years mixed up.

      •  Also, the BYU exit poll (none)
        conducted since 1982, has been right on the money consistently. Oh course, it's only conducted in the state of Utah where the demographics are probably easier to control for, hence less variation. But absolutely, exit polls have been pretty accurate in the past.

        Moderation, the noblest gift of heaven. - Euripedes

        by recentdemocrat on Sat Nov 13, 2004 at 08:47:48 PM PST

        [ Parent ]

    •  No, actually I'm in agreement with you (none)
      Exit polling was pretty accurate with some exceptions. I only included those few caveats to make sure that we didn't think that Fla 2000 was  the only problem.
         Fla was a unique set of circumstances. The networks tried to do it on the cheap, and VNS didn't properly allow for absentee ballots in Fla. They did no phone exit polling to get a handle on the size of the absentee vote and how it was going to break. It turns out they were off a few/ three percent and Bush had a higher total of absentee. That said, their prediction for Gore would probably have stood except for the high rate of spoilage, particularly Duval county. Also Palm beach ballot design, chads, etc... People thought they voted for Gore, told the exit pollsters they voted for Gore. Only problem was that the votes weren't always counted.
         There was a problem this year however. The new consortium NEP definitely screwed up somewhere. All the talking heads were signalling a Kerry victory by 8 pm eastern. Love to see the results of the investigations they're doing.

      Moderation, the noblest gift of heaven. - Euripedes

      by recentdemocrat on Sat Nov 13, 2004 at 08:37:12 PM PST

      [ Parent ]

      •  OK (none)
        It could be my reading comprehension skills aren't that sharp right now.  I thought I detected a bias toward recording the inaccuracy of the polls overall.

        Great job, by the way.

        •  Thank you. . The only other thing I was trying to (none)
          get across was not to trust the early afternoon and early evening leaked numbers. They are not predictive. The close of poll numbers have historically been pretty good, but then again, maybe the elections are getting closer, identifying problems with the weightings that weren't critical before? I don't know, but I'm in the camp we need exit polling.

          Moderation, the noblest gift of heaven. - Euripedes

          by recentdemocrat on Sat Nov 13, 2004 at 08:57:07 PM PST

          [ Parent ]

      •  Re: No, actually I'm in agreement with you (none)
        "That said, their prediction for Gore would probably have stood except for the high rate of spoilage, particularly Duval county. Also Palm beach ballot design, chads, etc..."

        Yup.  The funny thing about the Florida 2000 VNS "debacle" is that they were actually dead right about how people exiting the polls thought they'd voted.

        Between the unusually high spoilage rates in minority precincts and the Palm Beach Buchanan votes, Gore had the margin to win Florida rather easily.

    •  And this is why... (none)
      We can stress the inaccuracy of exit polls in these two particular situations (2000/2004) because these elections were much closer than any others since 1980. Think about it.  All the other elections since 1980 have had anywhere from 5% to 19% seperating the two candidates. If the margin of victory is well outside the margin of error on the polling, then of course the polls will be right.

      It gets much trickier when you have a very close race, like 2000 and 2004.  That's where the flaws in polling really tend to show up.

      "I don't want to bring politics into this, I'm just here for the drugs." Nancy Reagan, at a "Just Say No" event, 1985

      by Jack109 on Sat Nov 13, 2004 at 08:57:14 PM PST

      [ Parent ]

      •  That's also when... (none)
        ...the flaws in the election process itself show up.  Most people aren't particularly interested in whether Candidate X defeated Candidate Y by 19% instead of 14%, but they are extremely interested in whether Candidate X defeated Candidate Y at all.
    •  Re: You seem (none)
      "The problems started happening in 2000, as you mention, with the Bush campaign ... This is exactly when exit polling--which was very trustworthy up to that point--started to go haywire."

      But this isn't true.  The exit polls were wacky in every previous election as well.

      They're not trustworthy until you mix in the actual returns, and they never have been.

      "Without understanding that, you can't fully understand why some people are very suspicious about paperless voting machines and calling for fully counting the vote."

      You mix apples and oranges, and that's what I find so dangerous about all this exit poll nonsense.

      Paperless voting is a real threat.  It really has to be stopped.

      But this years exit polls have nothing to do with that.  

      And by basing the case for election reform on something that can be so easily disproved, the important case for real election reform is moved to the lunatic fringe.

  •  When all is said and done (none)
    Why the hell do we really need exit polling?  Or if we do, why does it have to be made public.  It serves a very narrow interest, and the public interest is not exactly high on the list.  I say no exit polling results announced until all the polls have closed nationwide. At least then it will not have any chance of affecting the election.
    •  Integrity and Honesty (none)
      If there'e enough discrepency between the exit polls and the actual tally, alarm bells go off. This comment will now be inundated by people saying that this happened this election. It didn't, the discrepencies are well within the 90% confidence interval more than 90% of the time.

      The world's address
      a place that's worn
      a sad pun that reflects a sadder mess
      In case you haven't already guessed:
      The world's a dress.

      by Jaiwithani on Sat Nov 13, 2004 at 08:42:10 PM PST

      [ Parent ]

      •  Nope (none)
        It didn't, the discrepencies are well within the 90% confidence interval more than 90% of the time.

        Three states: OH, PA, FL are well outside the 99% confidence interval.

        The chance of each was very small.  The chance of all three occurring together is astronomically small.

        There is no doubt in those three states:  either the pollster systemically screwed up their poll in a major way, or the vote tallying was wrong.

        It is not sampling error.

        --

        Timothy Klein

        by teece on Sun Nov 14, 2004 at 12:23:15 AM PST

        [ Parent ]

    •  In a word- Ratings (none)
      The exit polls help them to call the races. This year the networks were subdued because of 2000 and 2002, and didn't make any calls that weren't safe.I don't expect that caution to continue.
        The other reason is exit polls, properly done, can help prevent fraud. That's exactly what they did in Ukraine this year.

      Moderation, the noblest gift of heaven. - Euripedes

      by recentdemocrat on Sat Nov 13, 2004 at 08:43:20 PM PST

      [ Parent ]

  •  If voters fill out the survey cards anonymously... (none)
    ...why would they feel the need to lie about their choices?  I might not want to admit that I voted for Candidate X, but no one can know for whom any individual respondent voted.

    Even if the survey is conducted face-to-face, the respondent can be asked to flip a coin and not tell the pollster the results of the flip.  The respondent can then be asked to say that he voted for Candidate X if the coin turned up heads regardless of his/her actual vote, and to report his/her actual vote if the coin turned up tails.  At the end of the day, the pollster would subtract 50 from Candidate X's final percentage and multiply all percentages by 2.  So, if Candidate X received 74%, Candidate Y received 25%, and Candidate Z received 1%, we subtract 50 from 74 and multiply all numbers by 2.  Final figures for this particular location would be X 48%, Y 50%, and Z 2%.  The precinct figures are then weighted and summed to construct the state projection.

    Or am I reinventing the wheel here?

    •  Why do people lie? Don't know. (none)
       I'm  not even sure if Bush people were less likely to participate in exit poll this year. Something skewed towards Kerry, and NEP has some 'splaining to do.

      Moderation, the noblest gift of heaven. - Euripedes

      by recentdemocrat on Sat Nov 13, 2004 at 08:52:11 PM PST

      [ Parent ]

      •  Re: Why do people lie? Don't know. (none)
        "Something skewed towards Kerry, and NEP has some 'splaining to do."

        I don't think the exit poll performance this year was out of line with previous elections.

        And FWIW, the early afternoon numbers have skewed Democratic in the past 5 Presidential elections.

        •  Forget the afternoon numbers (none)
          They're indicative of nothing, IMHO. The problem was in the closing poll numbers. There was definitely a skew that showed Kerry leading comfortably. Remember all the talking heads going wink,wink, nod nod at around 8pm. Some of the articles I cited at the end go into much more detail. But even NEP has had to issue at least one preliminary report ( proprietary to subscribers naturally) to explain what happened.

          Moderation, the noblest gift of heaven. - Euripedes

          by recentdemocrat on Sat Nov 13, 2004 at 09:31:12 PM PST

          [ Parent ]

          •  Re: Forget the afternoon numbers (none)
            "Forget the afternoon numbers, they're indicative of nothing, IMHO. The problem was in the closing poll numbers. There was definitely a skew that showed Kerry leading comfortably. Remember all the talking heads going wink,wink, nod nod at around 8pm."

            The problem with the talking heads was deeper than that.

            The conventional wisdom over a 48 hour period up until around 9pm on election night was that it was Kerry's election.  The early afternoon exit polls and the closing exit poll numbers just fed into that already existing CW, and the talking heads ran with it.

            -------

            AFAIK, we don't know the exact closing exit poll numbers.  While I assume they had shifted somewhat from the earlier 51 -48 Kerry lead, for the sake of argument, let's assume they were still showing Kerry up by 3 nationally.

            If so, what's the problem here?

            I don't expect the exit polls to be correct until they start mixing in actual returns.  They never are.  They're not designed to be.

            The whole architecture of the Mitofsky-style exit polls is designed for the networks to be able to call non-blowout state races without having to wait for all the returns.  They're not designed to reliably pick winners at 7pm EST.

            In one of the articles you link, someone from Edison gives this quote:

            "No wrong projections (of winners) were made; the projections were spot-on.  The members used this data with sophistication and understanding of what data can and cannot be used for."

            He's not just blowing smoke when he says that.  It's what the Mitofsky-style exit polls are, and have always been.

            •  Nope (none)
              The final closing exit polls, weighted, all showed Kerry with leads in something like 9 of the 11 battle ground states.  Several were 3-5% leads.

              Yet the vote tallies shifted to Bush by huge margins, margins well beyond anything that can be explained by chance.  Either Mitofsky-style polling this election was horribly flawed, or the vote tallying was wrong.

              What happened this election is most certainly not the result of "normal" exit poll inaccuracy or chance.  That is quite easily ruled out simply by running the probability of this shift happening by chance.

              (It should also be note that exit polls can have quite large sample sizes, so the standard error is not necessarily big in these polls).

              --

              Timothy Klein

              by teece on Sun Nov 14, 2004 at 12:30:58 AM PST

              [ Parent ]

  •  Thanks. (none)
    A great summary, though I'm still thirsty for how exactly they do the weighting. I assume they just multiply by some correction for the number of people at that polling place relative to everywhere else.

    If so, they must not have done a good job choosing the precinct sample, or, things were a lot different than their older weighting models predicted.

    I have been extremely frustrated reading articles in the press where they say "Experts and analysts have debunked arguments based on exit polls" without any explanation of how they debunked it.

    If they only explained it with the depth you provided...

    Thanks.

  •  Excellent Overview (none)
    This is a wondeful overview.  I thought I'd make a few points.

    • The early exit polls suck every election cycle.  This is nothing new.

    • The weighted closing time exit polls are better than the early polls, but they are still not very accurate.  This data is not precise enough to be useful in non-blowout races.  Again, this is nothing new.

    • Exit polls are meant to be to be used in combination with actual returns.  This is the only time when they become precise enough to be useful in non-blowout races.

    So many of the conspiracy theories being peddled depend on folks not understanding the details of exit polling.

    The thing I find most fascinating is that when you press the conspiracy peddlers on the details, they begin to admit they don't care whether the theories are true or not - they just like the propaganda effect of claiming massive fraud.

    •  Nope (none)
      The weighted closing time exit polls are better than the early polls, but they are still not very accurate.  This data is not precise enough to be useful in non-blowout races.  Again, this is nothing new.

      This is simply not true.  You don't need vote tallies to make exit polls accurate.  You only need an accurate idea of demographic turnout and a large sample size.

      This is what statistics has been developed for, it can predict the election quite accurately unless it is very close unless the pollster really screws up or the vote tally is wrong.

      --

      Timothy Klein

      by teece on Sun Nov 14, 2004 at 12:35:07 AM PST

      [ Parent ]

      •  Re: Nope (none)
        "This is simply not true.  You don't need vote tallies to make exit polls accurate."

        I'm sure this is true in the hypothetical world you are imagining.

        However, in this world, it has been untrue in election after election after election.

        ---

        "This is what statistics has been developed for, it can predict the election quite accurately unless it is very close unless the pollster really screws up or the vote tally is wrong."

        Or unless the exit polls are explicitly designed to rely on actual voting returns in non-blowout elections...

        But, at least it's good to finally know what statistics was developed for.

        •  Sigh (none)
          I'm sure this is true in the hypothetical world you are imagining.

          However, in this world, it has been untrue in election after election after election.

          Think about what you are saying.  Once you begin to inter-mingle exit polls and actual voting data, you are relying an the soundness of the vote.  If the vote is suspect, the exit poll is suspect.

          Fact:  Exit polls are used quite successfully outside the United States to gauge the fairness of an election.  Germans use exit polls to call their week-long, manual election count, days before the official tally, and they have never made a mistake.

          Exit polls are a fine tool.  The weighted, closing exit polling data we have in this election is very far off the mark.  If the American exit pollster needs to use the actual vote tally to validate his polls, then the pollster is damn fool, and does not know his job (or is a complete cheapskate, and is underfunding the task).  We have all the stastitical tools we need to call an election with exit polling alone (assuming that our vote tally is not hopelessly FUBAR, which is a generous assumption).

          Those are the only two choices we have this election cycle:  the pollster is an idiot, or the vote tally is wrong.

          I, unlike you, am completely unwilling to just shrug and say "exit polls are dodgy."  Because there are many examples of them not being at all skewed.  This is a question we need to answer, not sweep under the rug.

          It is nowhere near as neat as you think it is.

          --

          Timothy Klein

          by teece on Sun Nov 14, 2004 at 12:52:38 AM PST

          [ Parent ]

          •  Re: Sigh (none)
            "Think about what you are saying.  Once you begin to inter-mingle exit polls and actual voting data, you are relying an the soundness of the vote.  If the vote is suspect, the exit poll is suspect."

            I agree with you entirely.

            "Fact:  Exit polls are used quite successfully outside the United States to gauge the fairness of an election.  Germans use exit polls to call their week-long, manual election count, days before the official tally, and they have never made a mistake."

            Perhaps they are not using exit polls that are designed to rely on actual election returns, the way the Mitofsky exit polls are.

            The Mitofsky exit polls are conducted by the networks to aid in calling state races early.  They were not designed to be of use in detecting fraud.

            "If the American exit pollster needs to use the actual vote tally to validate his polls, then the pollster is damn fool, and does not know his job (or is a complete cheapskate, and is underfunding the task)."

            Wrong on all counts.  You simply don't understand who is commissioning these exit polls, and for what purpose.

            "We have all the stastitical tools we need to call an election with exit polling alone..."

            Would these be statistical wrenches or statistical hammers.

            (As an aside, I find the people who have the worst understanding of political polling tend to be folks familiar with statistics who have no experience with polling.  Sam Wang, this year, was a laugh riot with pretty much everything he said.  The statistics crowd thinks polling is only about math, when it's actually half math and half something else.)

            If you'd like to commission an exit poll in 2008 that would be designed to catch fraud instead of calling races on TV, be my guest.

            It'd run you many many millions of dollars, and wouldn't be any use in an election this close, but I'd still sign your petition.

            •  Who (none)
              Perhaps they are not using exit polls that are designed to rely on actual election returns, the way the Mitofsky exit polls are.

              They are commissioned by the German media, just like the American exit polls.  I fail to see a reason why Americans would be so lazy as to have to rely on the vote tally, or are unable to fund the proper exercise, as the German's are.

              The Mitofsky exit polls are conducted by the networks to aid in calling state races early.  They were not designed to be of use in detecting fraud.

              They may not be designed to catch fraud, but that doesn't mean they won't.  But I am not talking about fraud.  I am talking about vote tallying problems, that may or may not be fraud.  Don't jump to conclusions.  There were lots of tallying problems in FL 2000, but not necessarily fraud.

              (Interstingly, that exit poll was right, Bush lost.  It was huge vote tallying problems that gave him the victory).

              But an abberation here or there could be shrugged  off, even if you accept the claim that the Mitofsky poll can't call a close election (which I don't, and we can verify that, as the methods they use are secret).

              But we don't have an abberation this election:  we have a systemic foul up across many states and counties.  If the exit poll just had some kind of sampling error, it would cut both ways: it did not.  The discrepancy was in Bush's favor every time, so many times as to be beyond the realm of just sampling error.

              --

              Timothy Klein

              by teece on Sun Nov 14, 2004 at 01:21:09 AM PST

              [ Parent ]

              •  Re: Who (none)
                "But an abberation here or there could be shrugged  off, even if you accept the claim that the Mitofsky poll can't call a close election"

                The Mitofsky poll is actually pretty good at calling close elections, precisely because it relies on the actual returns.

                ---

                I won't comment on the German exit polls, because unlike the Mitofsky polls, I don't have detailed knowledge of how they work.  And unlike you, I don't like to form dogmatic opinions about things I don't understand.

                ---

                "If the exit poll just had some kind of sampling error, it would cut both ways: it did not.  The discrepancy was in Bush's favor every time, so many times as to be beyond the realm of just sampling error."

                I'll try this one final time, and then leave you be, as you seem to have considerably more enthusiasm than either understanding, or the desire to achieve understanding.

                By design of the methodology, the Mitofsky exit poll numbers are not intented to be reliable before the actual returns are mixed in.  The numbers you cite as the "exit poll" numbers are not the final numbers.  They are not intended for forming the type of conclusions you have jumped to.  

                You are taking unfinished numbers and using to them to form erroneous conclusions.

                These unfinished numbers have been unreliable in every past election they were used in.

                The survey's methodology is all about producing good finished numbers, and it succeeds quite nicely.

                I'm sorry if you can't get your mind around these concepts.

                •  Over (none)
                  I won't comment on the German exit polls, because unlike the Mitofsky polls, I don't have detailed knowledge of how they work.  And unlike you, I don't like to form dogmatic opinions about things I don't understand.

                  Why do you have to start behaving like a child?  I have no dogma here.

                  By design of the methodology, the Mitofsky exit poll numbers are not intented to be reliable before the actual returns are mixed in.  The numbers you cite as the "exit poll" numbers are not the final numbers.  They are not intended for forming the type of conclusions you have jumped to.

                  To which I say bullshit.  I know that is what the claim is on the Mitofsky poll -- I have never seen any valid reason for doing it, and the pollsters won't explain.

                  But nevermind.  I'm through talking to you.  If you can't debate without getting childish, I'm done.

                  --

                  Timothy Klein

                  by teece on Sun Nov 14, 2004 at 02:02:22 AM PST

                  [ Parent ]

                  •  Re: Over (none)
                    "I have no dogma here."

                    Sure you do.  You have a faith-like devotion to the idea that the poll closing time exit poll numbers have some magical meaning that the pollsters themselves explicitly disavow.

                    You cling to this idea without supporting evidence because it leads to your preferred conclusion.  That's dogma.

                    "To which I say bullshit.  I know that is what the claim is on the Mitofsky poll -- I have never seen any valid reason for doing it, and the pollsters won't explain."

                    If you had some knowledge of the issues involved in polling, you'd understand the reasons intuitively.

                    Failing that, you could find the reasons explained explicitly, cogently, and repeatedly on this thread.

        •  Also (none)
          Or unless the exit polls are explicitly designed to rely on actual voting returns in non-blowout elections...

          If the pollster makes this assertion (that he needs votes to be sure of his prediction in something closer than 5%), then he is either: incompetent, lying, or being very cheap.

          But, at least it's good to finally know what statistics was developed for.

          Funny.  But think about it.  There is an entire field of mathematics devoted to making predictions based on random samples of a population.  We have actually gotten very, very good at it.  Life-or-death and multi-million dollar decisions are made all the time in this country using this science -- it works.  The primary difficulty of election polling -- figuring out who the electorate is going to be -- is gone on poll night:  the exit pollsters are talking to the electorate.

          There is no reason under the sun not to be able to get extremely accurate election predictions from exit polling, with the exception, perhaps, of cost.

          It is a complete cop-out to just pretend this is some really difficult to measure quantity -- it isn't -- or that we just got a bad sample this election -- we didn't.

          Again, either the polling firms employed by the media are so incompetent that they should be run out of business, or our vote tallying system is very screwed up.

          There is not a third choice here.

          --

          Timothy Klein

          by teece on Sun Nov 14, 2004 at 01:08:23 AM PST

          [ Parent ]

          •  Re: Also (none)
            "There is no reason under the sun not to be able to get extremely accurate election predictions from exit polling, with the exception, perhaps, of cost."

            There is no reason not to build a retractable dome over Minneapolis for use during the winter except for cost.

            ---

            And even if we had some kind of perfect exit poll that you fantasize over, it's still going to have a MOE that will render it useless for detecting fraud in an election as close as this one.

            "If the pollster makes this assertion (that he needs votes to be sure of his prediction in something closer than 5%), then he is either: incompetent, lying, or being very cheap."

            You fundamentally misunderstand the issue here.  It's not that the exit poll is so lousy that he needs the actual returns to be sure.

            It's that the survey is reliant on actual returns by design.

            I fail to see how this is so difficult to get your mind around.

            "Again, either the polling firms employed by the media are so incompetent that they should be run out of business, or our vote tallying system is very screwed up."

            No, it's neither of the alternatives you propose.

            The pollster is conducting his survery exactly as his client, the networks, have requested.  Doing it this way serves the clients' needs quite well.

            If you'd like a survery conducted for different purposes, I'm sure there are many pollsters out there you could hire.

            •  Yup, (none)
              Somebody is fundamentally misunderstanding the issue here, and I don't think it is me.

              You fundamentally misunderstand the issue here.  It's not that the exit poll is so lousy that he needs the actual returns to be sure.

              It's that the survey is reliant on actual returns by design.

              There is no good reason to design a poll this way.  Indeed, once he starts "correcting" his data, it is not clear that what he is doing is even valid.  The company does not and will not publish it's methods:  to be completely honest, I don't believe them when they say they design the poll to use vote tallies.  I say they are full of shit.  And they are unwilling to say what they are doing -- it is black magic, for all we know.

              On the MOE: totally wrong.  With a large enough sample size, it is possible to get the MOE down to very small sizes (easily under a 1%).  BYU does this every election.  MOE is not an issue except in super close states, like FL.  It is not an issue this election.  Most of the states that had odd shifts from Kerry to Bush show change outside of MOE.

              --

              Timothy Klein

              by teece on Sun Nov 14, 2004 at 01:58:44 AM PST

              [ Parent ]

              •  Re: Yup, (none)
                "There is no good reason to design a poll this way.  Indeed, once he starts "correcting" his data, it is not clear that what he is doing is even valid."

                He is not "correcting" his data.

                All pollsters do some form of weighting with their data.  The genius of the Mitofsky methodology is that it uses actual election returns to do the weighting.

                And if you want to argue that this weighting makes the Mitofsky survey not "valid" for detecting fraud or irregularities in a reasonably close election, you'd finally have said something that wasn't utterly inane.

  •  I am thrilled to hear all of this... (none)
    ...but I still have my doubts.  I understand that we've never really had such close elections when exit polling was used, so maybe we don't really know how accurate the exit polling is.

    BUT, it's hard to believe that with all the pre-election polling data (Zogby and Sam Wang both calling for a decisive Kerry victory), the historical information re: undecided breaking for the challenger, the anecdotal evidence of massive dem turnout, intangible indicators of Kerry victory (Nick kids vote, redskins, etc), and the FINAL exit polling indicating slight Kerry lead in Ohio, that Kerry didn't win.  I mean, what are the chances that all of that is wrong in a SYSTEMATIC way for Bush?  I'd believe it more if some of it predicted a Kerry win, and some a Bush win, but did anyone see any evidence of a Bush win from the available data?

    The only plausible non-fraud explanation I've heard that makes any sense is a staggering shift in the CW about republican turnout.  I don't believe that the undecideds "decided" to go almost 100% Bush.  That doesn't make sense, unless these so-called undecideds were actually Bush voters all along that had a hard time justifying their vote.  Anyway, the above explanation (increased Bush turnout) is a testable hypothesis, so WHERE ARE THE NUMBERS?  Can someone show me definitive proof that the Rovian 72-hour plan actually produced results?

    Until I see the numbers I remain skeptical and am more likely to say some sort of systematic disenfranchisement/fraud occurred.

    The problem with using the exit polls to shed light on it is that it becomes a circular argument.  The exit polls are increasingly refined to reflect the final results, but if the final results are skewed, then the exit polls can't tell us squat.

    •  Well actually the polls were all close (none)
       Remember Gallup LV had Bush up by 2, CBS LV had Bush by 1,ARG  LV had it a tie, Fox LV  had it Kerry by 1. Zogby made his call about 5:30 pm election day after he had some early exit poll numbers to match against his surveys.
         I think what you're seeing is the natural limitations that polls have when the race is so close. The undecided rule  says undecideds break 75-80% for the challenger. The converse of that is 20-25% of the time the undecideds will break for the incumbent.

        "Anyway, the above explanation (increased Bush turnout) is a testable hypothesis, so WHERE ARE THE NUMBERS?  Can someone show me definitive proof that the Rovian 72-hour plan actually produced results "

      What kind of numbers are you looking for that would show you increased turnout by Bush voters? I'm afraid I'm not understanding .

      Moderation, the noblest gift of heaven. - Euripedes

      by recentdemocrat on Sat Nov 13, 2004 at 10:05:07 PM PST

      [ Parent ]

      •  Republican turnout... (none)
        ...given US census data in 2000 v. 2004, registration numbers for states that keep track of party affiliation, etc.

        Also, look at it on a precinct-by-precinct level, compare 2004 vs. 2000 in absolute numbers of ballots cast.  If repub turnout is increased, shouldn't we see bigger gains in heavily repub precincts than in heavily democratic precincts?  Correct me if I'm wrong, but isn't that what the CW is, that this massive evangelical/rural vote turnout basically swamped any increase in what we were lead to believe was a massive Dem turnout in the metro areas?  LIke I said, the numbers should be available, does anyone have access to them?

        •  I sure as heck don't have them (none)
          But you make an excellent observation how to test the CW.
             From what I've seen posted North Carolina has precinct data but it's all intermixed into a hellacious file. I haven't seen anyone post that they analyzed the data the way you suggest.
             FLA  has a pretty good website but only to the county level. Don't know if each county has website breaking down further. We do here in California at individual county websites.  (Fla also has 'Dixiecrat' issue so no matter how you analyze the data some factions going to say it doesn't prove anything)
            Quite frankly, been so fascinated following the ongoing controversies that I haven't tried to do much independent research. There's at least three threads up now dealing with the voting. If you cross-post you just might plant the seed in someone to take on a project tomorrow. Just a thought.

          Moderation, the noblest gift of heaven. - Euripedes

          by recentdemocrat on Sat Nov 13, 2004 at 10:42:10 PM PST

          [ Parent ]

        •  Re: Republican turnout... (none)
          "LIke I said, the numbers should be available, does anyone have access to them?"

          Yes.  You do.  Precinct level election results are public information, and are available on the web.

          If you'd like to reinvent the wheel, and prove to yourself what happened on election day, rather than taking the word on non-partisan professionals who do this stuff for a living, all you need is a net-connected computer and some coffee.

          •  Well, I might just do that.... (none)
            And I'm judging that by the tone of your comment, this has been done by these same non-partisan officials?  Why isn't it published in a manner as to leave no doubt as to whether the CW explanation is correct instead of just saying, take our word for it?  Or am I missing something here?  Also, if it has been done, I'd love to see a link or something where I can look at their analysis.

            I mean, here's what I'm suggesting.  The votes for Bush or Kerry may or may not have been tampered with.  Thus, we cannot trust this data.  What we should be able to unambiguously trust is TURNOUT, i.e. total number of ballots cast, b/c that's independently verifiable.  Now, if we just look at say the top 50 dem precincts in some state, total their turnout, and select the top n repub precincts where n= a number such that the sum total turnout is the same as the top 50 dem precincts and then compare the ABSOLUTE change in turnout from 2000 to 2004, if the increased evangelical voter theory is correct, the absolute turnout increase should be greater for the R precincts than the D precincts.  If not, then something's fishy.

  •  The Source of the Misunderstanding (none)
    From the NEP FAQ:

    How are projections made?

    Projections are based on models that use votes from three (3) different sources -- exit poll interviews with voters, vote returns as reported by election officials from the sample precincts, and tabulations of votes by county. The models make estimates from all these vote reports.

    -----

    I've been baffled since the day after the elections that folks were convinced that the exit polls suggested fraud.  And I became ever more baffled as repeated debunkings failed to bring people around.

    I'm finally coming to the realization that it's lack of understanding of the above methodology that what makes folks unable to see what's going on here...

    •  I think one of the issues (none)
      is we see and have seen the networks project a winner as soon as the polls close, before any vote are tabulated. Not realizing that these states are well outside the margin of error ( blowout states) we have been conditioned (some of us since childhood) to accept the notion of the exit poll as gospel in ALL cases. From the little bit of research I did, I've gained a new respect for the potential limitations when doing it across a country as diverse as the US.
         With that being said, I'm still of the opinion that the exit poll data was skewing towards Kerry. Too many professionals were getting head fakes. Mitofsky and Edison had to have an late afternoon conference call warning people who do this for a living that there was a problem with the results and to exercise real caution with the numbers.
         The fact that no network made an incorrect call is not a testimonial to the accuracy of this particular exit polling exercise. This is the first year that these two particular firms have been running the process. ( Both companies have been in business a long time and Mitofsky himself is recognized as the 'father' of modern day exit polling, by way of clarification for others).
         When several seasoned reporters and editors who have had lots of experience with interpreting  and using exit polls had to change their leads, one is left to wonder how what in the data was reinforcing the sense that Kerry was headed to the presidency.
         FWIW, I think NEP needs to be more forthcoming with the data sets and use it as an education process for the American public and the media. It would go a long way to settling some controversies if there wasn't so much mystery surrounding what the numbers showed and when they showed it.

      Moderation, the noblest gift of heaven. - Euripedes

      by recentdemocrat on Sun Nov 14, 2004 at 06:53:34 AM PST

      [ Parent ]

      •  Re: I think one of the issues (none)
        "With that being said, I'm still of the opinion that the exit poll data was skewing towards Kerry. Too many professionals were getting head fakes. Mitofsky and Edison had to have an late afternoon conference call warning people who do this for a living that there was a problem with the results and to exercise real caution with the numbers."

        Yup.  The pre-mixed data was skewing Kerry, and NEP told their clients.

        But the pre-mixed data tends to be unreliable every year.

        And as far as the talking heads getting the head fakes, that has more to do with the fact that they're not up to speed on the details of how the Mitofsky methodology works - kinda like all the folks around here who still think those pre-mixed numbers indicate fraud or irregularities.

Subscribe or Donate to support Daily Kos.

Click here for the mobile view of the site