There has been considerable discussion in the past day or so about the meaning of the margin of error used by polls to estimate support levels for the various candidate. I have been saddened to see some real innumeracy in this issue, even among a professed social scientist who said that he works with statistics regularly.
A lot of people seem to be saying that even though Obama had a 5-point lead in the CNN poll, and the margin of error was 3 points, you could not say that this result is statistically significant, since 5 points is not greater than twice the margin of error.
These people are wrong. The same value can be used for the margin of error for the estimate of each support level as for the hypothesis that one candidate is ahead of the other. I'll show why after the jump. The basic thinking is that, when considering the difference between two estimated values, you can ignore one of the tails of the distributions.
Let's make this concrete. I am a mathematician, so I'm going to mathematize everything.
Let's say there are two candidates C1 and C2, and that a poll claims that C1 has a support level of p1 and candidate C2 has a support level of p2. And let's say that the polling firm says that the margin of error is espilon at a 95% level of confidence (eq. a p-value of .05).
Most people are with me so far, I hope.
Let's let p1 be the estimate of the true level of support for candidate q1, and likewise, p2 is an estimate of the true level of support, q2. (Ordinarily one would use the same letters and a hat ^ over the variable for the observed value, but I'll do it this way because I'm stuck in ASCII-land.)
The meaning of the margin of error (MOE) is this:
the probability that |p1 - q1| < epsilon is at least 95%.
(Alternatively, the probability |p1 - q1| > epsilon <.05)</p>
Similarly, the probability that |p2 - q2| <epsilon is at least .95.</p>
What a number of people are doing is using this logic. If |p1 - p2| > 2* epsilon, and p1 > p2 then the probability of q1 > q2 is also at least .95.
While this is true, this is not a tight bound. It is true that, if the difference between the two support levels is more than twice the margin of error, that it is easy to see that there is a significant result showing that the leader in the poll is probably the leader in real life.
The logic of the loose bound goes like this: for the person trailing in the poll to be ahead, at least one of the following events must be true:
E1: |p1 - q1| > epsilon
E2: |p2 - q2| > epsilon
Since the probability that either is true is at most .05, the probability that both are true is at most .05, and thus we have a significant result.
But the reason this bound is not tight is because the events E1 and E2 allow for variation that favors the lead! We are not really concerned about E1 and E2, but rather E1* and E2*
E1*: p1 - q1 > epsilon
E2*: q2 - p2 > epsilon
We can ignore the tails of the distribution on the sides that favor the lead. Now, since binomial sampling is (approximately) symmetric, if the probability of E1 is .05, then the probability of E1* is half that. Ditto for E2*.
So Probability(E1*) <.025 and Probability(E2*) < .025.</p>
Now we only see a bad polling result if at least one of E1* or E2* happens. We can bound the probability that both events have happened by the sum of the probabilities that each event happened separately.
I.e. Probability (q2 > q1) <Probability(E1*) + Probability(E2*) < .05</p>
So if p1 - p2 > epsilon, then it is a significant result.
What that means for the CNN poll: since the margin of error is 3 points, and the poll shows a 5-point lead, then we can conclude that the lead is statistically significant at 5 points. It does not have to be 6 points!