If you're on Kos, you'll have seen them: diaries and comments which tout a poll to show that either Hillary or Bernie clearly does better than the other when matched up against potential Republican nominees in a general election.
Usually, the poll in question is picked because it showed a particularly pronounced difference. If you're a bit skeptical, you'll have shared my wariness about these polling results having likely been cherry-picked to make the favored candidate of the diarist or commenter look especially strong. So what do the numbers look like when you review all recent polls? That's what I set out to discover.
To get a sizable enough sample of polls to counter-weigh their natural volatility (which is especially strong when measuring margins between candidates rather than candidates' individual percentages), I’m listing all the polls which include match-ups between both Hillary and Bernie against Republican candidates since November 1. And I'm including both national and state polls, though I'll list them separately.
As source for all these polls I've used RealClearPolitics. It's a conservative site, but the way they list the polls make this exercise a lot easier than how Huffington Post's Pollster site lists them. I did check to see if I missed any by going through HuffPo's feed of recent polls for any that said “Presidential GE”, though, and by checking the (woefully incomplete) Polling Report's page on national GE polls. Picked up at least one extra poll that way. If I still missed any let me know!
To ensure a scrupulous comparison, I’m listing only those poll results where both Bernie and Hillary were pitted against the same Republican. So if a poll pitted only one of Hillary and Bernie against Republican candidates but not the other, I do not list it; and if a poll pitted Hillary against, say, five Republicans, but Bernie against only two of those, I am listing only the results against those two Republicans, and omitting Hillary's results against the other three. Otherwise the resulting averages would not be fairly comparable with each other.
I've highlighted the very latest ones (from 2016), which is important because a trend break of sorts may be going on. If Bernie did not, in fact, do significantly better than Hillary in this kind of match-up last November and December (however upset pointing this out made the occasional Bernie supporter), he has certainly been doing better than Hillary in the polls that have come out so far in 2016.
In general it's a pity that relatively few pollsters ask these match-up questions, and fewer still ask them for both Hillary and Bernie. That means we are largely relying on two pollsters here -- PPP and Quinnipiac provide a majority of the listed polls. Considering how pollsters often have a "house effect," determined by their methodological choices, this necessitates a note of caution.
There are more of those, which I will try to cover below, but first: the data. I'll first try to paste them in as an image:
Too small to read? You can also just go and look at the data here.
Since I had to define advantages somehow, I have gone — rather arbitrarily — for describing a difference of at most 1% between Hillary's and Bernie's margins as No advantage, a difference between 1% and 4% as a Slight advantage, a difference between 4% and 7% as a Moderate advantage, and anything more than that as a Strong advantage. Your mileage may differ, and if you feel that those thresholds should have been defined differently I'm all ears. Before doing so do check out the section below about margins of error and how they might multiply when you compare the lead of one candidate over another and then the difference between the leads of two candidates in parallel questions. It suggests not to set these thresholds too low.
More below the story break.
Conclusions
Some of you will probably come to different conclusions about the above numbers than others. Personally, what struck me when I started writing this diary was that there wasn't all that much difference between how the two Democrats fared in these match-ups. For example, national polls in November and December didn't show any advantage for either Hillary or Bernie at all. There were eight of them, and taken together they were a wash. But then the new PPP and NBC/WSJ/Marist polls from Iowa and New Hampshire came in and I had to update my draft.
The most recent NBC/WSJ/Marist poll from New Hampshire showed a stronger advantage for Bernie than any other recent poll anywhere, and January's PPP poll from the state seemed to confirm that Bernie currently plays better than Hillary in that state — and is improving on his numbers. The UNH poll for CNN from a couple of days ago showed another whopping advantage for Bernie, but that pollster's numbers bounce about notoriously.
This month's NBC/WSJ/Marist poll from Iowa showed Bernie with an edge there as well, and the latest PPP poll from the state agreed, when the same pollster's previous polls in the state had shown a negligible advantage for Bernie in December and a Hillary advantage in November. In both states, then, PPP showed a shift towards a significant advantage for Bernie.
If there is a trend break, however, confirming it is complicated by the fact that only one new national poll has been released this year which asked general election match-up questions for both Bernie and Hillary (and even then only pitted both Dems against Trump). It was conducted by NBC/WSJ, and while it showed a moderate advantage for Bernie, up a bit from their previous national poll in November, past NBC/WSJ polls seem to have shown greater competitive advantages for Bernie than other contemporary polls (see below). So it would have been nice to see another national poll with this kind of data. Or two.
It's also complicated by two state polls with GE match-up data which appeared just a few days ago. A PPP poll from North-Carolina showed no difference in how Bernie and Hillary perform in that state, and no change in how either of them performed from PPP's previous poll in that state. And a poll by Florida Atlantic University actually showed Hillary doing better in that state against the Republicans — in much the same way a previous poll from Florida found last November.
Back in November, other state polls from non-early primary states showed a moderate advantage for Bernie in Colorado and Wisconsin, and no advantage for either candidate in Michigan. All of this may may feel plausible (Bernie doing better in heavily white states in the upper Midwest and West, while Hillary matches up more firmly in Southern and Great Lakes states), but with public perceptions of candidates likely to change a lot in those states once the Iowa and New Hampshire campaigns peak, it's all very tentative.
Not so much difference?
If you take the average of all the match-ups, against all Republicans, both nationally and in individual states, since November 1 — a total of 93 match-ups from 24 polls — you get a Hillary lead of 0.4% and a Bernie lead of 2.8%. To my mind, that difference is minor, especially when you take the somewhat bewildering variation between individual polls into account. After all, according to one recent poll from New Hampshire, Hillary lead Cruz by 8 points; according to another, she was behind by 4. According to one recent national poll, Bernie lead Trump by 15 points; according to another, he was behind by 2. If different polls show the Democrats performing so very differently against Republicans, how meaningful is it if Hillary or Bernie does a few points better in any one of those polls? The house effects of different pollsters might be creating greater differences than the impact of whether Bernie or Hillary is posited as Democratic nominee. In its defense of the value of current polling even when random samples are hard to get by, the British Polling Council argued that “if a well-designed, representative survey finds that the public divides 70-30% on an issue, then a margin of error of even 10% cannot alter the fact that one view is expressed far more widely than the other”. But that's obviously not the kind of margins we are seeing here.
Similarly, the differences between how the different Republicans fare against Bernie and Hillary appear, when you look at this average, to be significantly larger than those between the two Democrats. Trump does worst, while Rubio does best. If you take the average of all match-ups listed here, Trump is behind the Dems by an average of 4.3, while Rubio is ahead by 1.3. But that mashes together polls which asked about all five of the Republicans and polls that only asked about Trump, or only about three or four of the five. If you only look at the polls that asked about match-ups between each of Trump, Cruz and Rubio and both Democrats (which leaves us mostly with PPP and Quinnipiac polls), Trump's behind the Democrats by an average of 5.1 and Rubio's ahead by an average of 1.4. This mixes up national polls and polls from individual states, so those numbers can't be taken as reflection of their actual, national leads, but they illustrate how they fare compared with each other. And the difference between Trump and Rubio is two to three times as big as that between Hillary and Bernie.
Or a significant advantage for Bernie?
The numbers look very different when only looking at the polls from this month, though. In that case, you get the same average Hillary lead of 0.4% — but an average Bernie lead of 8.1%. That's a significant difference, and certainly suggests that Bernie's comparative advantage over Hillary in these match-up polls against Republicans has increased in the new year.
Those are averages from a total of just eight polls though (with 30 match-ups), seven of them state polls, most of which conducted in Iowa and New Hampshire which are arguably demographically favorable to Bernie — the two state polls which weren't from either of those states also didn't see any advantage for Bernie. Still, they include three PPP polls and a national NBC/WSJ poll which we can directly compare to previous polls by the same pollsters. In three of those four cases, there is a shift to a greater comparative advantage for Bernie, which you might interpret as meaning that Bernie's attractiveness is improving as news of the race starts reaching more people. More polling data sure would be helpful though!
Going back even further
When observing a seeming shift to a greater advantage for Bernie, one thing I thought would be interesting was to go back even further to see how he was doing in these match-up polls back in August and September, when Hillary's primary numbers took a temporary dive. In other words, does the improvement from November/December to January constitute a long-term trend of Bernie becoming more attractive, or was he doing much better last September as well, and were the months of November/December in fact more of a temporary dip?
I traced back national match-up polling to August last year. The only Republican candidate who was systematically included in general election match-up polls from back then until now is Donald Trump, so I've limited the below chart to how Hillary and Bernie have done in polls against him.
As you can see, pollsters have drastically disagreed on how the Dems would fare against Trump, with Emerson College showing some of the worst numbers for the Democrats and NBC/WSJ and McClatchy/Marist some of the best. When comparing how Hillary and Bernie fared against Trump, there does seem to have been a bit of a current, with Bernie doing noticeably better in three polls back in September/October, Hillary doing better in three polls in October/November, and Bernie again faring better in the most recent two polls. Such a trend would run conveniently parallel to Hillary's fortunes in the Democratic primary. Pollsters seem to have different house effects with these numbers though. NBC/WSJ is the most bullish on Bernie's competitive advantage, while Quinnipiac seems to ‘lean Bernie’ as well in this particular context; PPP and Emerson College have been the most bullish on Hillary's advantage.
A major caveat: polls do not predict the future
Although I love myself some polls, there is one important thing to keep in mind in all of this, and that is that match-up polls this far from the election are far from conclusive evidence about how candidates will actually fare. Much is still likely to change in how various candidates are viewed by the electorate at large as the primaries proceed; and the general election campaign itself of course can also still markedly change a nominee's fortunes.
This is where arguments like "Bernie would do better than the polls suggest once more people get to know what he's really like" or "Bernie would start doing worse if he's the nominee once the Republican machine starts going after him" come from. Such arguments can be used to dismiss inconvenient evidence, of course. But to some extent they can be legitimate: the polls only show how Bernie/Hillary as they are viewed now would fare against the Republicans as they are viewed now. In reality, once the fall campaign gets under way, those perceptions will change. In fact, 538's Harry Enten went back 70 years in the polls last November to find that general election match-up polls a year out from the elections were off on the lead from the actual election result by an average of no less than 10.6% (though one would hope that January polls were already a bit closer).
A good example of how such changes can take place over time is how Carson started off with very favorable match-up numbers when compared with those of other Republicans, which have eroded significantly since thanks to critical media coverage. If Bernie supporters who argue that ‘their’ candidate has been a victim of a media blackout mean it, they should also take into account that when more extensive media coverage comes, this may include a lot of negative coverage as well. That might have an impact on his numbers among the many voters who are as of yet only fleetingly familiar with Bernie, especially outside the universe of Democratic primary-goers. Vice versa, even after so many years in the public eye Hillary's favorability ratings are hardly immune from ups and downs either, and there's no guarantee that the general election campaign might not create another sea change in her public image, for better or worse. Media coverage of Hillary seems to have become more positive since the first Democratic debate and the fiasco of the Benghazi hearing, when compared to the previous period of all-emails-all-the-time, but that might change again, especially once the Republican attack machine gets into gear.
More caveats: multiplying margins of error?
There is also a more technical note of relativity - and I hope I am explaining this correctly, because my knowledge of how polls work is wide but shallow, so please help me figure this out if I'm wrong. Pollsters promise that they have each candidate's percentage of the vote correct within the margin of error 95% of the time. But the margin of error doubles when looking at the lead of one candidate over another. If, say, Bernie is at 51% of the vote plus or minus a margin of error of 3%, and Cruz at 46% plus or minus 3%, then this means that the difference between them could reasonably still be anything between 54% Bernie / 43% Cruz and a 48% Bernie /49% Cruz. That is to say: the poll would find a 5% Bernie lead, but the margin of error for that lead would be plus or minus 6%.
Now a lead being within the margin of error doesn't make it meaningless, of course. The existence of a margin of error doesn't mean “that each possible number in that range has an equal likelihood of being right, far from it in fact," DKE commenter OGGoldy explained the other day. If a poll “says you're at 51%±3% it is much more likely to fall on 50, 51, or 52, than it would be to fall on 48 or 54.“ It's just that the chances that it actually is a lead drop below 95%, Trosk added. There is apparently a formula to calculate the probability that one percentage is higher than another, but that's all way out of my depth.
Now things get more relative still, because in this diary I'm comparing the leads of two candidates against various Republicans with each other. So what would the margin of error be on that comparison? Twelve points? Does it really work that way? Theoretically it would seem so: take the latest NBC/WSJ/Marist poll from New Hampshire. Clinton leads Trump 45% to 44%, with a margin of error of plus or minus 2.3%. So Clinton's 95% likely to be at 43-47% and Trump at 42-46% - a four-point span. That makes for a margin between the two candidates of somewhere between Clinton +5 to Trump +4 - a nine-point span. Sanders leads Trump by a whopping 56% to 37%, with the same margin of error, putting Sanders theoretically at 54-58% and Trump at 35-39%, making for a Sanders lead of between 15 and 23 points. Which means that Sanders does better than Clinton against Trump by anywhere between 10 and 27 points, if you want to play it safe and stick to a 95% chance that the numbers are right - a seventeen-point span. Doubling margins of error.
I'm not exactly sure this is actually how this works, though. When I asked the good folk in the comments section of the Daily Kos Elections digests to weigh in, an interesting conversation ensued, even if some of it was a bit beyond my grasp, and one commenter explicitly warned against “summing the error bars”. But it's clear that comparing the leads of two candidates against the same opponent, when each of those leads already involves a MoE that's twice the stated one, just adds more uncertainty — let alone comparing the average leads of two candidates against a bunch of opponents!
And in addition to all of that, this still only addresses the statistical likelihood that the result of a poll of a truly random sample of voters is off, when pollsters' main problem these days is that getting a genuinely random sample of voters is all but impossible, and weighting results to emulate a random sample is more art than science.
It's almost enough to make this entire exercise seem meaningless, but where would the fun be in that?