Originally posted at Overdetermined
When we started this site, we never meant to let our partisan identification get have anything to do with what we wrote about. After all, when writing about data, voter files, polling, journalism, microtargeting, Linux and other such things, you'd think that there would be plenty of material to write about. And, well, there is, but to my eyes, the perpetrators of stupidity in polling are mostly on the otherside.
Today's lesson comes from that bastion of truth-seeking and truth-speaking integrity, the Editorial Pages of the Wall. St. Journal. Known parrhesiast Stephen Moore decides to show us how not to read a poll.
There's more.
Moore is writing about a poll that the good people at Target Point conducted on why people voted the way that they did in the Presidential election. Neither the dataset, crosstabs, sampling methodology, research methodology nor the questionnaire are available. From this story, all that we know is that n=1,000, and Moore declines to link to any further information on the poll. At the time of writing, there is nothing on the Target Point website about this poll, either. Most importantly, I would want to know for whom this poll was conducted.
Why does this matter? As I've written before (here and here), universes matter, and so does the client. The client has has goals that they're trying to accomplish, and they hire a pollster to conduct the research and build a strategy for them by asking selected people questions. So, if this was a poll for the Republican National Committee of their top 1,000 donors, the results would be very, very different from a poll of 1,000 people from Milwaukee, WI, or 1,000 Latinos from the Sunbelt, or 1,000 people who well represent the nation at large. Given that Gage and Lundry do good work, and are credible, I'm going to assume that this was a poll of 1,000 people who well represent the nation at large for a client with no message to test and no goal to accomplish, and who wanted pure research.
The important thing to note is that I have no reason to make that assumption. Moore has not provided any details of the poll, and neither has the pollster. This is a perfect lesson in how not to conduct journalism on polling. But, really, who's surprised? It's the Editorial page of The Wall St. Journal.
This is just off the top, without even getting into the details. This is already bad journalism, and not worth reading, but since a lot of people do read him and it has influence, let's address what Moore actually said.
That's a big problem because even though 84% of voters say they are center or right on the ideological spectrum, the 48% in the middle, i.e., independents, are tilting heavily toward Democrats. The fairly narrow victory by Barack Obama in the popular vote disguises an "enthusiasm gap" among Democratic and Republican voters. Some 65% of Obama voters "strongly supported" him, whereas only 33% of John McCain voters "strongly supported" the Arizona Republican. This helps explain the river of money for Mr. Obama and the massive grassroots advantage for the Democrats.
This is already a gold mine of bad analysis. The first big one is in the first sentence: 84%. Without getting too deeply into formal logic, the key term in this claim is "or". Let's assume that there are two operands in this proposition: "center" and "right". If the number of people who are center is 83% and the number of people who are right is 1%, that 84% proposition is as true as it would be is the numbers were 33% and 51%, respectively. So long as those two numbers add up to 84%, they can be absolutely anything, and there's no reason to assume that those numbers are remotely equal or close to each other. (Not to fixate, but if we had the crosstabs, this wouldn't be a problem.) He's relying on the fact that we're going to be impressed by the big number and the way that we process the word "or" in natural language to make us think that the right are in a natural majority in this country. It's slippery and sneaky, but there's no doubt in my mind that that's what he's up to.
Now, come on, shouldn't I be fair to Moore? He does say that 48% of people are independents. That means that 36% of people are right, which would mean that the left in the country would be 16%. 16% Left, 48% Center and 36% Right. That seems to fit what we imagine to be the truth about America, right?
Wrong. Wrong. Wrong.
Moore is either stupid or disingenuous, and I'm leaning towards the latter. He's conflating two separate things and relying on the confusion. The left/center/right question measures ideology, and is completely subjective. The Democrat/Independent/Republican question measures party identification, is not completely subjective. In other words, it's possible to be a centrist or even a righty and be a Democrat, or be a centrist or even a lefty and be a Republican. (Granted, this is changing as the parties become more polarized, but it's still true.)
Here's the more important thing, though. I joked about the meaninglessness of the ideology question before, but now let's really address it. Why is this a useless question? Part of it has to do with the subjective nature of the question, but a big part of also has to do with the relativity of what's being measured. Let's address the first part, the subjectivity. When you measure someone's party identification, you're measuring whether or not he's a registered member of a party, votes in a party's primaries, donates to their campaigns, volunteers for campaigns, etc. There are external and verifiable qualities to this, and you are measuring to what degree a person posseses them or how many of them he possesses. When you ask the ideology question, you are asking someone to describe how he thinks of himself, which does not necessarily correlate to any observable external properties or behaviors. A respondent could quite easily imagine himself to be quite liberal, but vote for the conservative party. Moreover, let's examine the relativity of what's being measured. Whether it's on the liberal/moderate/conservative scale or the left/center/right scale, people's ideological self-identification tends to function as a comparison to the people around them. Here's an easy example: compared to the people around him, Chet Edwards is pretty moderate guy. Compared to people in Manhattan, he's pretty conservative. GIven that he's from Waco, and not the Upper West Side, if you asked him about himself, he'd probably answer that he's a centrist or a moderate. Now, if we take this seriously, consider what happens when we ask an ideology question for 1,000 respondents in a nationwide sample. The people who answer "conservative" in Oklahoma are pretty different from the people who answer "conservative" in New Jersey, but they'd still get lumped into the same category for analysis, which would render moot the whole point of cross-tabular analysis.
Of course, as I said, given that Target Point didn't start polling two minutes after falling off the turnip truck, they know all this. They know that ideology doens't transpose one to one with party identification or political behavior, and I really doubt that they said so in their report or briefed Moore with that. This is most likely all Moore. This leads us to the important question: is he stupid or just a liar? I'll settle for both.
Anyone who questions, by the way, that ideology doesn't transpose to political behavior should examine the cases of Susan Collins, Olympia Snowe, Gordon Smith, Lincoln Chafee, Chet Edwards, Byron Dorgan, Heath Schuler or any other elected official who represents a constituency that is not composed of 50%+1 vote of the ideology of her party. Whether or not it's the case that each of these politicians is sufficiently charismatic, benefits from incumbency, or whatever, it is the case that each of these politicians draws from enough people across the ideological spectra to get innto office.
Now, the seeming fault in my logic is that these politicians are also drawing from people who are not of their party, but here's my rebuttal to that:
- Party identification is an objective measurement of something tangible, so we can use it, and
- Party identification is a stronger predictor of behavior than self-reported ideology.
In short, ideology questions don't give us anything really useful for predicting political behavior, and pretending that it does means that you're either an idiot or a liar. Given that this is the case, I'm more inclined to believe that Moore is the idiot or liar than Target Point.
DD