There are many sources of error in polls. One source of error is that only a sample of the voting population is asked -- that's the statistical error. The size of the sample can predict how likely any size of statistical error is. The current jargon for that is "Margin of Error" or "MoE." People who don't know statistics, especially broadcast journalists, have some weird ideas about that. Here are some facts.
1) The MoE is a distance beyond which the statistical error is unLIKELY to go. It is not a limit beyond which it cannot go.
2) Something narrowly within the MoE is somewhat more likely than something narrowly outside the MoE, but that does not mean that it is likely.
3) The MoE is a distance from the measured value that the polling organization gets. That isn't the distance from the value that they report. If they have a sample of 1001 and they report that your guy got 45%, they may have measured that anywhere from 446 to 455 saying that they planned to vote for your guy. A report of 45% for Smith to 46% for Jones can be a spread of 446 for Smith to 465 for Jones; it can also be a spread of 455 for Smith to 456 for Jones. Obviously, the first pair of results from the sample is more likely to represent a real difference in the population than the second pair of results is.
4) The MoE represents the distance from the measured value which gives some low probability of the sample mean being farther from the population mean. What low probability? That is something that the polling firm chooses. It depends on the size of the sample, but two polling firms with the same sample size might choose to announce different MoEs.
For specifics on these generalizations, follow me over the jump.
A "bell-shaped curve" goes on forever, at least in principal. The way that you differentiate a narrow bell-shaped curve from a wide one is a measure called the "standard deviation," or "SD." Now for an evenly split population, the size of the sample determines the SD for the error of the sample mean from the population mean. (And we're assuming that the population mean is 0.5.) The likelihood that the sample mean will be within one SD from the population mean is 0.6826; the likelihood that the sample mean will be within 2 SDs from the population mean is 0.9544. Thus, the likelihood that it will be more than 2 SDs away is 0.0456.
Now, for poll shops and anyone else doing sampling, an error on either side is a problem. We don't worry about that, though. So, in the following list of polls, I'm only showing the probability of an error in one direction from statistical causes. (I got the data from the RCP site. I've omitted the results and the pollsters name. All I'm interested in showing is the relationship between MoE, SD, and likelihood of error from statistical causes.)
N |
SD |
MoE |
MoE/SD |
Prob. |
1109 |
1.501 |
3 |
1.998 |
2.28 |
1500 |
1.291 |
3 |
2.323 |
1.02 |
751 |
1.825 |
3.5 |
1.918 |
2.74 |
2721 |
0.959 |
2 |
2.087 |
1.83 |
1112 |
1.499 |
3.4 |
2.268 |
1.16 |
800 |
1.768 |
3.5 |
1.980 |
2.39 |
1000 |
1.581 |
3.1 |
1.961 |
2.50 |
"N" is the sample size (number of people polled) reported in the poll. "SD" is the standard deviation of the distribution of samples of that size around the mean of the population if that mean were 0.5. "MoE" is the margin of error that the polling company reports. "MOE/SD" is the margin of error expressed in standard deviations. "Prob." is the probability -- expressed as a percent -- that a sample would deviate from the population in one direction
from statistical causes more than that margin of error.
So, we see that the reported margin of error is something like 2 standard deviations.
Note:
Among the non-statistical problems which polls face is that various subpopulations may differ in their ability to be reached by pollsters from their likelihood to vote. One way to deal with this is to overweight respondents from difficult-to-reach populations. That totally fucks up the statistical analysis. Everything I'm writing assumes that this isn't how the polling firm deals with that problem.
Plenty of people think that the size of the sample should increase with the size of the population. Actually, that doesn't give much advantage. For an example, look at this table with a ridiculously small sample, 5. The second line is the likelihood of a getting a particular number of red marbles from the limit of the sampling without replacement as the size of the population increases without limit. (The first line is the number which you will get from the sample.) The last line is what you get sampling a population of 100. This shows how little the size of the population matters.
0 |
1 |
2 |
3 |
4 |
5 |
0.03125 |
0.15625 |
0.31250 |
0.31250 |
0.15625 |
0.03125 |
0.02814 |
0.152947 |
0.31891 |
0.31891 |
0.152947 |
0.02814 |
Of course, the population is not actually split 50-50. If it is close to that, the SD will be very near to the calculated value. (If it is far from that, the MoE doesn't matter).
One actual problem comes from situations where the two candidates who get your interest have, in total, much less than 100% of the responses. The proper statistical response is to consider the sample size for calculating SD (and, hence, MoE) for the total of those responding for the 2 candidates. If the poll's sample is 1,000, 300 say Smith, 340 say Jones, 200 say Brown, and 160 are undecided, then:
2 SDs for the sample as a whole is 3.16.
2 SDs for the part of the sample who say either Smith or Jones is 3.95.
The lead of Jones over Smith is just outside the MoE if the second method is used. Of course, the real question is whether the supporters of Brown will be disappointed and choose one of the leading candidates and -- if so -- which one.