Let me preface this diary, by saying that I am a statistician by day (and political obsessive by night). I love fivethirtyeight and Nate because that’s the language I speak on a daily basis. My days are filled with looking at data, inventing ways of testing data, and figuring out what statistics means. I deal with lots of data, millions and millions of datapoints, on a regular basis. And so, I have this warning: WE DON’T KNOW WHAT WE THINK WE KNOW! Follow me below the fold for an examination of error and bias and why we need to redouble our efforts.
Polls and statistics get things wrong all the time. Dewey defeats Truman, the Bradley Effect, the Shy Tory effect, are all fine examples of polls and early samples getting the results wrong. What I intend to do in this diary is explain, in layman’s terms, how some of these problems may arise, and why even Nate’s wonderful fivethirtyeight will even get things wrong. More generally, this diary is a call to arms, providing inspiration to redouble our efforts as we get closer to the election.
In this diary, I will deal with two main problems for statistical analysis: error and bias. Error is our inability to get exact answers to what we are trying to estimate. The plus or minus margin in polls is one measure of error. Wikipedia once again provides information for statistical error. Error is also called statistical noise, stochastic variation, uncertainty, etc. Bias, on the other hand, is actively getting something wrong because an effect we aren’t modeling. The Bradley effect is a great example of a statistical bias not an error.
Should we be worried about errors?
The error variance on any of the polls is important to consider. Models like Nate’s or the pollster average are going to have smaller error variances than any individual poll. The error variance on a poll is a function of the sample size. The larger the sample size, the smaller that error variance is, and the better our estimate is.
However in statistics there’s a problem called multiple testing or multiple comparisons. Basically, when we conduct many experiments or take many polls, we are likely to observe a ‘fluke.’ Put another, our chance of observing an outcome is a function of the chance of that outcome and the number of times that we conduct our poll or experiment. Why is this important? Well, if we have 20 or 30 states that are battlegrounds, then our polling data is likely to be outside the 95% margin of error for at least one of those states. So if Obama is ahead by more than the margin of error in 20 states that does not guarantee that he will win all of those states. This doesn’t mean Obama is going to lose CA on a fluke (that would be a once in 100,000 or more occurrence). What it does mean is that none of these states are in the bank, so volunteer, phonebank, and GOTV. If we get ‘unlucky’ in two important states (say PA and VA), then the election could well be over.
What about bias?
Bias, on the other hand, is a much nastier problem for polls to approach. Bias comes in many different forms. Acertainment bias is when we have a sample that is not representative of the population. Much has been written on the cell phone problem: 1, 2, 3. However, pollsters may overestimate this effect, and so Obama’s numbers are better than they appear. Let’s take a guess that 1-2 percentage points is falsely being gained here.
Similarly, the Bradley effect may be present. There’s a reasonable debate as to whether it exists, but let’s assume that it exists as well. We could stand to lose another 1-2 points here.
If we shaved 3-4 points off Obama’s total and added it to McCain’s then the race would be a tossup. I’m not saying this is the case, but I am suggesting that some biases and a bit of bad luck in the error variance can yield a McCain victory. This is not the lock we necessarily think it is. We should be afraid of that victory. We should be out hitting the streets, the phones, and contributing.
Get involved now because this isn’t something that we should leave up to chance.