Going back through some older diaries, I saw a feel people deal with variance - which was very well done - but a bit overly complicated in explaining the formulas and such. I'll skip the formulas and use something that's a little easier to understand - the coin flip.
After the fold...
The Coin Flip - Variance
If we were to flip a coin 500 times the odds that
- the number of heads and tails would be within 1% of each other is 20% (or 1 in 5 times)
- the number of heads and tails would
not be within 4% of each other is 35% (or about 1 in 3 times)
If we were to do 2 sets of 500 coin flips, the odds that
- the number of heads and tails on both sets would be within 2% of each other and within 1% of an even split is 4% (or 1 in 25 times)
- the number of heads and tails on both sets would be outside the 4% range is 12% (or 1 and 8 times)
What Does This Tell Us?
Well, if we assume that a 4% (or greater) lead in a poll is conventional wisdom for the cutoff between a close race and a (somewhat) significant lead then it tells us that
- 17.5% of the time Heads has a significant lead over Tails
- 17.5% of the time Tails has a significant lead over Heads
- 20% of the time Tails and Heads are in a (an almost) dead heat
Of course, neither Heads or Tails is "ahead", if we flipped the coin another 500 times on November 4th both would have the same likelihood of coming up more.
What it really tells us though, is that even if polling companies have true random samples (or perfect weighing of the variables), the polls are going to be significantly wrong quite often.
Obviously, polling companies don't have true random samples or perfect weighing of the variables.
Variables
This has been diaried in depth here and elsewhere before, so I'll just list some of them without much explanation.
- Reaching A Representative Amount Of The 9 to 5'ers, 5 to 1'ers, and 1 to 9'ers
- Determining Who Will Actually Vote
- Is Each Candidate's Potential Voters As Likely To Respond
- Is One Candidates's Potential Voters More Likely To Not Be Home
- Reaching Cell Phone Only Users Or Proper Weighing If You Don't
- Handling Of 3rd Party Candidates
- Dealing With Undecideds And Leaners
- Etc...
What Does This Tell Us?
Well, it tells us there's a whole bunch variance and variables - and that any poll - even from the best company - shouldn't be given too much weight.
The Bottom line
If you see a poll (or even several) that make you want to jump in glee or toss your shoe at the computer - go ahead - but it's a wee bit silly.