Here at Daily Kos Elections, we pay a lot of attention to polls. Part of that is paying attention to the errors in polls, so that we can tell the difference between normal poll-to-poll variations and changes that actually mean something. So we were excited to see a clever experiment using a Florida poll that showed just how much error can come from the way polling data are processed.
Most people are pretty familiar with random error, which gives rise to the famous margin of error stated in nearly every telephone poll. This error alone can cause a decently sized poll to bounce around quite a bit, and if the sample is too small, as is routinely the case in the infamous UNH polls, the margin between two candidates can jump around like a jackrabbit on crack.
But there’s many more sources of error in polling, from how the respondents are contacted to how much support a third party candidate has. The study in question looks at variability resulting from processing data after interviews have been conducted: how to weight the demographics of the poll, and how to identify likely voters.
The New York Times Upshot, in partnership with Siena College, conducted the poll of Florida. Then they gave all the data to four other pollsters, shown above, and they each chose a different combination of valid methods to weight and screen the data.
The results? Hillary Clinton’s advantage ranged from +4 to -1.
Conclusion: Excluding random error, differences in the margin of five points between different pollsters are normal. Including random error, even larger differences are still normal for valid methods, and should not cause panic. In the end, it comes down to the same old advice: watch the polling averages.
For full details, I strongly recommend reading the original article.
47 days remain until the election. Click here to make sure you're registered to vote. And while you're at it, make sure your family and friends are registered too.