Americans are inundated with political polling results on a regular basis, and when actual elections are held, the results often deviate from the poll predictions. Why is that?
I am not a social scientist, but I was exposed to the basics of polling when I was in college 50-odd years ago. Using random statistical methods to sample a large population is a well established area of study developed over centuries. However, the techniques that were developed and taught to generate reliable polls have become increasingly difficult to adhere to in the 21st century.
First, let’s discuss the questions themselves. Words and phrases should be chosen to be neutral and avoid emotional triggers. There are inherent biases in the way questions are asked. Pollsters are human and may not even recognize their own biases. I’m not talking about “Do you still beat your wife?” types of biases. There are subtle unintended biases in the way questions are phrased. For example, order matters. If people are asked “Do you prefer A or B?” they will get a different answer than if they ask “Do you prefer B or A?” Asking a question early or late in a poll can also result in different answers. Then there is the explanation conundrum. Do you start your question with some basic factual information, or just allow your subject to answer based on their preconceptions? Imagine the difference in responses if you precede the question “Do you believe age is an important factor in a Presidential candidate?” with the statement “Donald Trump is 78 years old, Joe Biden is 81 years old.” Is there implicit bias in just presenting that factual statement? In a well designed poll, the same questions are asked in multiple different ways to try to eliminate those kinds of biases. Reputable pollsters take extreme measures to eliminate biases, but if a nefarious pollster wants to skew a poll to generate a favorable or unfavorable result, it’s pretty easy to phrase your questions to get any result you want.
Now consider the sample, the people who will be answering the poll questions. If you want your poll to reflect the general population, it’s critical that demographics be considered. Geography, age, gender, race, nationality, religion, culture, education, income — all these factors will influence the results and must be reflected in your poll sample. The fraction of women and men, executives, high school dropouts, college graduates, hourly workers, boomers, millennials, Hispanics, Blacks, Caucasians, atheists, Hindus, Christians, and Muslims in your sample should reflect the same proportions in the general population.
The demographic constraint leads to the first rule of sampling, ALWAYS PRE-SELECT YOUR SAMPLE. The people you will be polling should be known ahead of time and should be selected to ensure the correct demographic distribution. In an ideal poll, if your expected response rate is 25 percent and you need 1000 samples (more later), then you must pre-select 4000 people to call or send your questionnaire to, and you must never supplement those 4000, regardless of the responses. Then for example, if your poll response is dominated by white, Christian, male boomers with no college degrees, with few representatives of other minority demographics, then you can weight the responses by demographic category to boost under-represented responses and suppress over-represented groups. This of course is dangerous, amplifies statistical uncertainties, and should be applied sparingly.
One consequence of this rule is that a high quality poll must never allow self-selected participants. Anyone who wants to participate is automatically disqualified. Otherwise your poll will be contaminated by activists with strong opinions, which won’t represent your population. (Note that this automatically means that internet polls are ridiculously inaccurate and unreliable.) I often point out that my family was once selected to be a Nielsen family. Nielsen knew and liked our family demographic and so we received a box that reported our TV watching habits. When our year was up, my wife begged to be allowed to continue because we were having so much fun with the process, but of course, simply by asking to continue, we disqualified ourselves. I suspect no families are ever allowed to repeat, but I am not certain.
The pre-selected sample requirement has changed over the years because the increased use of cell phones with caller ID means that fewer people are answering and response rates have dropped. As a result, polling samples have been allowed to expand by asking for demographic information at the end of the poll and adjusting the distribution accordingly. But you can’t cover every possible demographic category, so that means that the granular control over demographics that is possible with pre-selection becomes much more problematic.
What about sample size? The rule of thumb is that the margin of error (the degree to which your sample responses reflect the whole population) is 1/sqrt(sample size), so if your sample results in 1000 total valid responses, then your MOE is 1/sqrt(1000) or 3.2 percent. One hundred valid samples corresponds to 10 percent accuracy, 10,000 responses reduces the error to one percent. One thousand responses appears to be the holy grail — fewer responses result in unacceptable inaccuracy, but more responses are very difficult and time-consuming to achieve.
Finally, let’s discuss the most fraught aspect of polling — determining the likely voter. Even the most reliable poll with truly unbiased questions, random sampling, and a large sample size can only tell you how your population feels about issues and candidates. But not all of those people will actually cast a vote. The likelihood of voting must be assessed by the pollsters, presumably accounting for voter motivation by demographic category, and is a major source of error in predicting election results. GOTV efforts or endorsements by respected celebrities (!) can skew likely voter statistics for underrepresented demographic groups. Single-issue voters may be more or less motivated in a given election cycle. In fact, the increased likelihood of voting by younger, suburban females post-Dobbs is destroying normative assumptions about motivation.
The bottom line is polling is hard, is getting harder, and is easy to screw up. Polls can be accurate or unreliable depending on how well biases are controlled and statistical methods are applied. Please remember that when you read the next poll results and take comfort or consternation from the huge uncertainties that inflict the polling process.
