In August, Tennessee state Rep. Justin Jones won back his seat in the legislature after getting expelled by the Republican majority for protesting gun violence. His margin of victory was an overwhelming 56 percentage points, in a district Joe Biden had won by 41 points in 2020 and Hillary Clinton had carried by 36 points in 2016.
This is the sort of overperformance that’s easily dismissed as a one-off, with such unusual and dramatic circumstances that it couldn’t possibly hold any broader meaning for the country at large. And that’s true—indeed, no single special election should be dissected in order to divine the will of the electorate.
But some 300 miles away, at the far eastern end of Tennessee, in a ruby red and nearly all-white district, virtually the same thing happened on the same day.
And then a month later, there was a repeat in the district next door.
In fact, there have been 27 typical special elections pitting a Democrat against a Republican in the 2023-24 election cycle so far, and Democrats have overperformed Biden in 20 and Clinton in 23. When numbers like that start to pile up, it’s time to sit up and take notice.
And what we notice is this: It’s beginning to look a lot like 2018 around here. That’s very good news.
We track House and state legislative special elections in which there’s exactly one Democrat and one Republican running. The only caveats, which wind up excluding very few races, are: The two candidates must combine to garner more than 90% of the vote, and they both must appear on the ballot with their party labels.
So far this year, Democrats in special elections have been doing an average of 7.6 points better than Biden’s margin in 2020 in the same districts and 12.0 points better than Clinton’s margin in 2016. Since Biden won the national popular vote by 4.5 points, and Clinton won it by 2.1 points, that translates to a political environment with Democrats running 12.1 points ahead based on comparisons to Biden and 14.0 points ahead based on comparisons to Clinton. Averaging the two values gives us a figure of D+13.0.
What does a similar exercise give us for 2017 through October that year? D+13.0. You can see each special election plotted against Clinton’s performance in each district below, with 2023 on the left and 2017 on the right.
Peering into the crystal ball
The resemblance is striking: Right now, things look as they did during the 2017-18 cycle, which was a very strong year for Democrats. But is it too early for the numbers to mean anything?
We already know that special election results, averaged over an entire two-year political cycle, can give us solid intel about the political environment (see our recent post on this correlation for details). But very interestingly, the figure at the top of the post (repeated just below) shows that this correlation appears to work well even when we have data for only half of an election cycle:
The light orange circles show calculations based on special election results up until October of the year before general elections for the U.S. House—that is, each odd-numbered year, dating back to 2013. (These are the years for which we have district-level presidential numbers that we can use as a basis for comparison.) The dark purple circles are the subsequent values for the House popular vote in November of the following year.
We can see that the values calculated from special elections give us a decent indication of what the future holds. Typically, they’re about 3 or 4 points higher than the actual result, which is a little larger than the difference we get if we use a full cycle’s worth of data.
The current value for special elections is, as noted above, about D+13. Now, that doesn’t imply that Democrats will win the House popular vote by 13 points next year, or even that the margin would be 13 points if the election were held today. However, the numbers are very good news nonetheless. Put another way, even if Democrats’ ultimate margin in the House vote were half of the current figure, they’d likely retake the House in 2024.
Common objections to the use of special election results in a predictive capacity simply don’t hold water in the face of the evidence. You’ll often see cherrypicked data—for example, using just a single special election—to claim that specials don’t mean anything. And it is true that any one special election doesn’t mean anything by itself. But that’s why we average them all together.
Averaging also takes care of the problem of unusual circumstances and low turnout resulting in extraordinary results. Outlier special elections can easily see 50-point overperformances for one party or another. These will typically be balanced out by outliers for the opposite party, but even if not, when averaged with dozens of other results, they have only a marginal effect on the final numbers.
In reality, you can pretty much see what’s coming down the pike a year ahead of time. In one sense, this is surprising because it makes it seem like all that effort and drama of campaigning doesn’t make much difference. However, a lot of the decisions about the election cycle are made in the year prior, based on the observed environment at the time—who’s retiring, who’s running, who’s raising money—and these decisions do indeed make a difference in the final outcome (see, for instance, the effect of an open seat in another predictive tool that Daily Kos Elections has developed, the House Vulnerability Index).
Things that don’t work, Part A: Odd-year generic congressional ballot polls
We don’t have a 30-year track record for partial cycle calculations based on special elections, as we have for full-cycle values. But a five-year record is not exactly nothing, either. And it’s far, far better than other measures we have available. Two that we might consider are polls of the generic congressional ballot (that is, surveys asking voters whether they plan to vote Democratic or Republican in for the House) and election results from Virginia, an often-competitive state that holds gubernatorial and legislative races in odd-numbered years.
Those we do have a 30-year track record for, and congressional ballot polls prior to election year, for starters, are simply terrible:
That’s … not very helpful. The numbers aren’t completely wrong 100% of the time: You’ll notice three dots near the diagonal line that indicates an on-target prediction. However, they are wrong often enough—all of those other dots much farther from the diagonal—to be completely untrustworthy.
Here’s another way of looking at it. The two lines plotted below just don’t really resemble one another:
But polling methods are evolving, and we have more polls today than 30 years ago, so what about the most recent data? Here’s a graph of the last five cycles in the same style as the graph at the top of the post:
The odd-year polling average varied from just D+3 to D+6—a very narrow band—while the actual House popular vote ranged from R+5.7 to D+8.6. Not very helpful.
Things that don’t work, Part B: Virginia off-year elections
Off-year elections are important in their own right for obvious reasons, but Virginia elections, in particular, are often taken to be an important bellwether, possibly because the national press is based in and around Washington, D.C. Unfortunately, these results can often be quite misleading.
If we use the popular vote for Virginia’s lower chamber, the House of Delegates, and compare it to the previous presidential cycle, we get massive swings due to changes in which sets of seats are actually contested. Often, the direction of the swing from the prior election is informative, if not the magnitude. But even then, it’s far from consistent, meaning there’s no way to determine ahead of time which years will provide useful information and which will not.
In the years prior to midterm elections, Virginia holds elections for governor. These races provide more robust data since every voter in every part of the state gets to participate (whereas some legislative districts inevitably go uncontested). Still, there have been some big misses.
Overall, Virginia elections are sometimes helpful, sometimes not—kind of like a supersized special election. The governor’s races, at least, yield more information about the following year’s political environment than any one special election, but less information than special elections as a whole.
How the special election numbers change
What should we expect to see going forward? Here’s how the calculated political environment, based on special elections compared to presidential results, has looked for the past three cycles and the current cycle:
Note that the time scale starts in March of the odd year; prior to that point, the small sample size yields wildly fluctuating averages. But you can see there’s simply not a whole lot of change after October of the off year. Part of that is because each new election is averaged in with the older results, so it takes some really dramatic changes to drag the whole average in a new direction.
Two events could change the calculated number going forward (but not necessarily the underlying environment). First are the special elections held in November concurrent with Election Day, which in 2018 brought down the numbers. Another is the possibility of primaries with lopsided turnout favoring the GOP, if any specials take place concurrently with Republican presidential nominating contest (assuming, of course, that there’s any interest remaining in the Republican primaries).
But the rest of the story is that the underlying environment only rarely changes, even in the face of, say, a pandemic and global economic shock. Indeed, the bounciest of the three completed cycles shown—2018—turns out to be so bouncy because of the vagaries of which states held special elections when. When that is accounted for (because of factors unique to the 2018 cycle), the trends almost completely disappear.
In recent years, there was one exceptional instance when we’re pretty certain there was a significant change in the electoral environment—following the Supreme Court’s evisceration of abortion rights in 2022. Yet it barely registers in the data, in part due to the small number of specials held after the decision in Dobbs v. Jackson Women’s Health Organization was handed down.
And yet, despite this sea change, special elections correctly predicted the overall 2022 electoral environment while simultaneously alerting us to the effects of the Dobbs decision—a paradox we discuss in a separate piece. It’s unlikely we’ll see a repeat of this behavior this cycle given that it’s a presidential cycle, but we’ll keep our eyes out.
What to watch
So what should we pay attention to going forward?
1. Keep an eye on the special election averages. Don’t worry about short-term trends in general. However, a large shift in special election numbers combined with a plausible mechanism for the shift may indicate a change in the electoral environment.
2. Beware both-sidesing. Stories on special elections will often feature “both sides” commentary: a Democratic operative saying special elections are going great and a Republican pooh-poohing the data. Data cannot be both-sidesed. The data (currently) looks great—and historically has been predictive.
3. Likewise, ignore gripings about how “low-turnout elections don’t mean anything” and “special elections are focused on arcane local issues.” It’s true that special elections can be very strange, especially when they engage few voters. But that’s why you average a bunch of them together. Again: The data, historically, has been predictive. End of story.
4. Treat November 2023 general elections as additional, not supplanting, data. Don’t ignore Virginia elections! Or any of the other elections. But don’t treat them as the be-all and end-all either.
5. Do not pay attention to generic ballot polls for 2024—yet. Polls are very valuable and can give us all sorts of useful information, but they cannot predict voting preferences a year ahead of time—likewise with presidential polls. (Remember the August 2015 polls with double-digit Clinton leads? Now scroll back up to the top and see what 2015 specials had to say.)
6. Politics isn’t physics. You can’t calculate political trajectories with the accuracy of a ball falling in a vacuum. So we can’t pinpoint the results of next year’s election today. But politics also isn’t a random number generator. Special elections have predictive value for a reason: People actually going to the polls to cast votes means something.
As of today, we have numbers showing a great electoral environment for Democrats. And in the recent past, these numbers have not moved dramatically over the remainder of a cycle. There’s plenty of work to be done, but for now … things look good!
Related posts:
Special elections are closely correlated with the House popular vote: Still true, six years later
Dobbs changed everything in 2022, yet the political environment remained the same. How can that be?
