There’s been a certain amount of confusion surrounding 538’s forecasting this primary season. The site’s bewildering array of widgets and articles containing a mix of modeling and punditry certainly don’t help matters. Here I’ll be looking specifically at 538’s Democratic state primary forecasts based on polling averages (i.e., their bread and butter) and ignoring everything else.
The models
The primary forecasts (plus a couple caucuses) are built upon two polling models, polls-only and polls-plus, that are themselves built upon weighted averages of individual polls:
- The polls-only model is based only on polls from a particular state; for example, only polls of New Hampshire are used in the New Hampshire forecast.
- The polls-plus model is based on state polls, national polls and endorsements. (National polls are used in a slightly unusual way; they’re a contrarian indicator. More about that later.) The polls-plus model also seeks to account for how the projected results in Iowa could affect the results in New Hampshire and how the results in those states could affect the results in subsequent contests.
The two models tend to converge as election day approaches, but as we’ll see there are some differences in their aggregate performance.
Please note that fifteen of the contests held so far lacked sufficient polling near election day to meet the forecasting threshold so I haven’t included them. This may seem like an obvious choice, but I’ve seen people try to knock the forecasts by bringing up stale and/or sparse polling data from states that didn’t make the cut.
Some people are all about the win percentages. I only care about this little doohickey:
Detailed polling forecast
This delightfully toggly widget distills the models into best estimates for Clinton’s and Sanders’ performances, complete with a set of probability curves and confidence intervals. Pick a state and watch the curves jump around. It’s kind of mesmerizing.
Now, on to the results.
Polls-only
Polls-only scatter-plot
The polls-only forecast for 23 states predicted a delegate-weighted average lead of 17.6% 19.18% for Clinton. The actual delegate-weighted lead was 18.63%, a difference of only -1.02 0.55 points. There’s some variance in individual forecasts, as can be seen in the scatter-plot above, but 90% of the polling errors fall between -14.3 points (favoring Clinton) and 10.1 points (favoring Sanders). The correlation coefficient is roughly 0.96. It’s a solid result however you slice it.
Polls-plus
Polls-plus scatter-plot
The polls-plus forecast predicted an average lead of 19.52% 21.28% for Clinton, deviating from the actual result of 18.63% (referenced in the prior section) by only 0.9 2.65 points. Interestingly, there’s less variation in polls-plus, with 90% of the polling errors falling between -9.2 and 10.5 points, and a correlation coefficient of about 0.97.
Does polls-plus have a slight Hillary lean and polls-only a slight Sanders lean? Perhaps, but both models are so close it’s basically splitting hairs.
Both models fared exceptionally well in the five most delegate-rich states that have voted so far: NY, TX, FL, IL and OH. Michigan has been the only big miss when you look at the actual delegate impact. Yet even though MI is in the dataset stinking up the place, regression to the mean applies and the results balance out on the whole.
So what does it all mean?
538’s primary polling forecasts are pretty accurate. So are other polling averages. It’s better to toss all the available data into a big pot and stir well than to overthink the individual results. Surprises can certainly happen but their impact tends to diminish over multiple races.
Even with their flaws, these math-based tools handily beat the cherry-pickers, unskewers, and pundits. Anyone wanting to prognosticate without embarrassing themselves should make good use of them.
Update 4/28/16: Corrected a calculation error in the delegate-weighted polling averages. This change does not have a meaningful impact on the analysis.
