It seems that every election season produces its share of amateur election forensics purporting to reveal massive fraud. A recent effort in this genre, apparently by two psychology graduate students (Axel Geijsel and Rodolfo Cortes Barragan), asks, “Are we witnessing a dishonest election?” [Link fixed.] The paper offers a bold answer that almost assures it a certain audience:
Is [Clinton’s] claimed advantage [over Sanders] legitimate? We contend that it is not, and suggest an explanation for the advantage: States that are at risk for election fraud in 2016 systematically and overwhelmingly favor Secretary Clinton.
If that seems overconfident, in an online appendix, the authors assert that they aim to “encourage more people to be critical of ‘reality’” and to promote “a free and open discussion about the potential for fraud in American elections.” One might say that they’re Just Asking Questions.
There is no a priori reason why their questions couldn’t have been good ones. But, no, this is just another exercise in bad election forensics. Do I say that because I am trying to suppress free and open discussion? Nope, I’m freely and openly expressing that the work is poor. If you wonder why, read on.
(For purposes of this analysis, I mostly will rely upon the authors’ data as made available online here. I won’t attempt to address every point, only what I take to be the most important ones.)
tl;dr update
If you take exit polls pretty seriously, maybe this is all you need to know: The authors find that in primary states with exit polls, the vote counts show that Clinton did 14 points worse in states with paper trails than in ‘paperless’ states. They think this is highly suspicious. The authors also find that, in the exit polls, Clinton did 15 points worse in the paper-trail states than the paperless ones. So, if they trust the exit polls, presumably they should conclude that the gap is legitimate. But they don’t seem to have noticed.
The case for fraud
Geijsel & Cortes Barragan (G&CB) begin by comparing Clinton’s delegate shares in two groups of states: those that “that feature voting procedures wherein the accuracy of electoral results of a primary ballot vote are backed by a paper trail,” and those that “do not have such a paper trail.” Of the 31 primary states in their analysis, they code 18 as having paper trails, and 13 as not. According to them, those 13 paperless states are Arkansas, Delaware, Florida, Georgia, Indiana, Kentucky, Louisiana, Mississippi, Pennsylvania, South Carolina, Tennessee, Texas, and Virginia. Clinton’s average delegate share in those states (about 65%) is 16 points larger than in the other 18 states (49%). Wow, how about that? (If you’re thinking, “Srsly?!”… yeah, you have a point.)
By the way, I just gave away the one weird trick that accounts for the pie chart above. The chart doesn’t actually mean that Sanders got 51% of delegates in paper-trail primaries: instead, his average delegate share in those primaries was 51%. In that calculation, his 100% share in Vermont weighs the same as his 44% share in New York. Actually, Clinton won a majority (52.6%) of the delegates in these states. But you have to admit that a pie chart that shows Sanders Winning! is more impressive, in its ways.
You may notice two things about the list of states without a paper trail. One is that most of them are southern states. G&CB’s appendix asserts in passing that they find the pattern “across the U.S., not just in the South,” but they do not seem to have given the matter much thought. (Hmmm, is there any other reason why Clinton may have done better in these states?)
The other thing that might strike you is that several of these states actually do have paper trails for many or even most of their voters. In Florida, for instance, although many counties use ‘paperless’ Direct Recording Electronic voting machines as their “accessible” equipment, most voters use hand-marked paper ballots that are counted by scanners. So G&CB’s basic categories are problematic. Still, some of these states are almost entirely ‘paperless,’ so let’s go with that for now.
Their test of alternative explanations
Now, the short paper asserts, “In the Appendix, we show that this relationship holds even above and beyond alternative explanations….” The most intelligible test comes first. G&CB use multiple regression analysis to control for two variables that might account for the relationship between paper trails and Clinton vote share. (Neither of these measures is included in their online data.)
- First is the proportion of non-Hispanic whites in the state (not in the Democratic electorate). (The implication here is that all other racial/ethnic groups can be treated as indistinguishable.)
- Second is what G&CB call “blueness,” which they describe as “a weighted average of the number of times each state had voted for the Democratic presidential candidate in the elections between 1992 and 2012.” The authors do not explain why they control for “blueness,” why they chose this measure, or exactly what the measure is. We can think of it as related to the premise that Sanders did better among more liberal primary voters.
G&CB report that % of non-Hispanic whites and blueness both are strong predictors (Clinton did worse in states with more non-Hispanic whites, and in bluer states) — but even controlling for these variables, Clinton does about 9 points worse in states with paper trails, and this difference is statistically significant. [Incidentally, the statistical tests implicitly treat primary states as if they were a random sample, which is conventional, albeit rather odd if you pause to ponder it.]
Better controls obliterate their result
This analysis is unconvincing on its face. If you’ve read this far, it probably occurred to you a while ago that it would be helpful to control for the proportion of blacks, not just for the proportion of non-Hispanic whites. You may also have thought that counting Democratic wins in the last six presidential elections seems like a crude way to measure “blueness.” So, as a sanity check, I tested alternative controls:
- Using Kaiser Family Foundation calculations based on the March 2015 Current Population Survey, I controlled separately for the proportions of Hispanics and of non-Hispanic blacks in each state. Based on abundant polling and reporting, I expect to find — and I do — that the proportion of non-Hispanic blacks is an even stronger predictor of Clinton vote share than the proportion of Hispanics.
- I use Obama’s vote share in 2012 as a straightforward measure of “blueness.” (I don’t expect the results to be very sensitive to what “blueness” measure is used. For what it’s worth, my t value is similar to theirs.)
These controls aren’t wonderful. I would rather control for the demographic characteristics and liberalism of Democratic primary voters — but I would have to rely on crude exit poll data, where available. It is easy to think of other controls to add, but with only 31 states in the analysis, we need to be wary of overfitting.
So, what happens when we use these alternative controls? The paper trail “effect” blows up. Specifically, using these controls, the estimated “effect” of paper trails is just -1.3 points with a standard error of 3.4, t = -0.39, p = 0.70 — indistinguishable from random noise.
What about exit polls?
A section of G&CB’s paper reports that in primary states with exit polls, Clinton did about 3.1 points better in the vote counts than in exit polls. (Here is a 2008 diary I wrote to rebut some myths about using exit polls to check vote counts.) As the authors say, the logic of their analysis suggests that the discrepancies should be “even more exaggerated in states without paper voting trails.” Indeed, one might expect that the exit polls were spot on in paper trail states, and far off — at least 9 points — in non-paper trail states.
Yeah, no. Astonishingly, G&CB’s reported results indicate that the average (mean) exit poll discrepancy was only 2.75 points in states without a paper trail, and 3.41 points in states with a paper trail! These average discrepancies are statistically indistinguishable, and the tiny difference is in the opposite of the predicted direction!
How did GC&B miss this, and how do they construe their results as supporting their thesis of fraud? Instead of reporting these differences, they report “effect sizes” based on the variability (standard deviations) of the discrepancies in each group of states. The discrepancies in the paper trail states happen to be more varied — mostly because of one very large discrepancy, in Arizona — so the “effect size” is smaller. This approach may seem sensible to psychologists who generally work with abstract measures — but these measures aren’t abstract. A 3-point average gap between exit poll results and vote shares is a 3-point gap, regardless of how varied the vote shares are.
(That large discrepancy in Arizona does influence the mean for paper trail states. We can use medians instead: the median discrepancy is 2.9 points in states without a paper trail, and 1.55 points in states with a paper trail. At least this 1.3-point difference is in the direction they expect, but again it could easily be noise.)
More ado about nothing
So, not only does a slightly more elaborate version of their best analysis reveal no statistically significant difference between so-called “paper trail” and “no paper trail” states, but their exit poll analysis also shows these groups of states to be essentially indistinguishable. Paper trails don’t seem to affect or ‘predict’ the results at all. The authors’ support for their heroic claim that Clinton’s lead over Sanders is not “legitimate” crumbles upon critical scrutiny.
Remember, Geijsel and Cortes Barragan contend that their main purpose was to encourage “a free and open discussion about the potential for fraud in American elections.” In the abstract, I support that goal. Much can and should be done to protect U.S. election systems against fraud, error, and mischance, and to demonstrate that election results are valid. As part of that effort, voter-verified paper ballots or audit records should be used in every state, and should indeed be routinely audited. On some level I wish I could believe that impugning the validity of Hillary Clinton’s primary victories on a flimsy pretext will help to improve U.S. elections. But actually, I think Clinton deserves better — and I don’t think this gambit can work anyway. Conditioning people to associate election integrity concerns with sore-loser CT probably isn’t a winning strategy.