For reality-based observers, 2018 is shaping up to be an ugly year for Republicans. While we don't usually give advice to the Republicans, we do have one suggestion for them: ignore everything.
But for the rest of us, there are lots of reasons to pay attention to special elections—in concert with other indicators such as polls and the generic congressional ballot, of course.
What’s so special about special elections?
The magic of special elections is that they can predict the future. Not with a high degree of specificity, but we can identify well ahead of time whether the November elections will be Good for Democrats, Bad for Democrats, or somewhere in between, if we have enough special elections to work with. And this year, the numbers are unmistakably saying Good for Democrats. Certainly we can’t expect every Democrat to perform 20 points better than Hillary Clinton or Barack Obama did in their state or district. But some will, even while others underperform.
By comparing special election results to previous election results in the same district for both state legislative seats and House seats, we’ve created a composite value called the Special Election Index. This Index, which is analogous to the average overperformance compared to presidential results, is currently showing the best environment in 30 years for Democrats. But more importantly, it tracks with the House popular vote in the following November election:
This approach isn’t perfect—1998 had poor-quality data for one—but there’s a very strong correlation nonetheless:
Generally speaking, the House popular vote is usually a few points higher than the Special Election Index value. Remember, though, we can’t make an exact prediction, just a pretty good one.
Why not just use congressional special elections? Wouldn’t that work better?
Why not? Let’s try it. Here’s the same graph as above, but with just congressional special elections.
Hmmm. That isn’t too pretty.
It might be better to compare congressional special elections to the previous presidential election instead. But you can see that graph here, and it’s even worse.
So that’s your reason. Using just congressional elections simply doesn’t work very well, almost certainly because there’s not enough data.
There’s a certain logic to the idea of using just congressional specials to predict congressional elections, though. After all, many voters tend to vote for a different party further downballot, and state legislative special elections can feature very low turnout, very small districts, and very strange circumstances. We deal with this in part by filtering out some of the weirdest ones (here’s our inclusion criteria list), but the rest of those issues are solved simply by averaging out the oddness.
Here’s the same type of graph as above, but this time using just special elections in state legislative seats. You can see it’s a much better correlation. The best, however, is the combination of state legislative specials and congressional specials, shown in the first section.
But those state legislative special elections are sO low-turnout ...
A reasonable concern. How can super-low-turnout elections mean much of anything? And doesn’t the fact that the best Democratic overperformances are seen in some of the lowest-turnout special elections mean we can’t learn much about the high-turnout midterms?
No, because the worst Democratic underperformances are also seen in some of the lowest-turnout special elections. True, there does still appear to be a trend (not always a significant one though) when you look at all the data. But after accounting for a couple of other important variables, there is no longer any hint of a correlation to be seen between Democratic performance and turnout:
But what happens if you ignore this and assume the trend is there anyways? That’s a great idea, if you’re a Republican! Please proceed! But if you want to know what’s really going on, that’ll lead you down the wrong path and result in some pretty poor predictions. We showed some examples here.
But they have such huge swings ...
How can elections with such giant, random swings in the margin (as compared to presidential results) be relevant to the midterms?
First, these swings are not so random after all; they’re mostly predictable. Two factors can account for most of the variance in margin swings compared to presidential margins: the presidential swing in the district, and the presidential margin in the state the district is in.
Second, the swing in any one special election isn’t too concerning;. What we care about is the average, and as the first section shows, the average lines up quite nicely with November results.
Finally, midterm congressional elections also have large swings. They’re not as prevalent, for sure, but they do happen.
Here’s the swing for each 2014 congressional race, on the left, and the swing for each special election in the 2014 cycle, on the right.
Red circles are seats held by Republicans in 2013, and blue circles are seats held by Democrats. House races don’t just blindly follow presidential results. First, many such races are considered so uncompetitive that one candidate campaigns little (or not at all). Even those races that are strongly contested don’t line up nicely, however. And the effect of incumbency is clear. Nonetheless, the ranges of margin shifts are pretty similar for both congressional and special elections.
But here’s where it gets interesting. Remember, we’ve been looking at average swings in special elections. And the average swing from the 2012 presidential margin in the 2014 congressional elections was -7.9. The average swing for special elections? -7.4.
But what about Virginia?
The Virginia governor’s race, a high-profile, high-turnout, regularly scheduled election last November, showed Democrat Ralph Northam overperforming Clinton’s margin by just 4 points, nothing close to the average 12-point overperformance we’ve seen in special elections. For that matter, the margin in the New Jersey governor’s race was almost the same as the 2016 presidential margin. Does these mean these races, which were held under conditions more similar to the midterms than most special elections, are telling us 2018 might be a bit underwhelming for Democrats?
No, not at all. Because both of these races in open seats are entirely consistent with the behavior we’ve seen in special elections in states that Clinton won, in districts that did not swing towards Trump. These two factors would predict Democratic victories of 8 and 17 points in Virginia and New Jersey respectively, very close to the actual margins of 9 and 14 points. The simultaneous legislative races in New Jersey and Virginia were also, on average, closer to predictions than special elections were. More details can be found here, but there is nothing about the Virginia and New Jersey elections that tells a different tale than the special elections.
Republicans, however, are welcome to pretend otherwise.
Special elections resources
Data:
Results compared to presidential margins
Special Elections Index, and results compared to prior legislative race margin
All Daily Kos Elections data
Posts:
Introduction to Special Elections Index
Correlation of Special Elections Index with House popular vote
Uncertainty and predictions using special elections data
Turnout and special elections
Special elections swing is correlated to presidential swing and presidential margin
Special elections trends with time and turnout, and consistency with statewide elections