Skip to main content

(Cross-posted at the Princeton Election Consortium)

Many readers of this site know that pollsters vary in their methods. However, existing solutions, such as correcting for bias, have attracted controversy. As a result, one prolific pollster can dominate the discussion.

Here I present a solution based on a very simple -- and neutral -- statistical principle. If adopted by Pollster.com, TalkingPointsMemo, and FiveThirtyEight, it would make poll aggregation far more useful.

It is well known that pollsters vary in their methods. The American Association of Public Opinion Researchers has established common standards of practice and encouraged transparency, driven in part by Mark Blumenthal of Pollster (the Princeton Election Consortium's data source). But poll-sniffers aficionados know that Rasmussen Opinion's results consistently trend more Republican than other organizations.

I should note that this problem is not limited to Rasmussen. The recent Pew survey showing a 10-point lead for Obama is also an outlier. In principle, a good solution would do something constructive with both of these sources of information.

Differences like these present a challenge to poll aggregators. An obvious solution is to estimate the size of each pollster's bias, then subtract it. However, this generates three new problems: (1) Who is the neutral reference point? Gallup? Quinnipiac? Rasmussen? (2) What to do about pollsters who do very few polls? (3) What if the pollster changes methods mid-season?

For my Meta-analysis I have chosen a simple solution that gets rid of most of the bias: use median-based statistics. Here's how it works. 

Imagine the two following similar sets of poll margins between candidates A and B:

Data set 1: A +2%, A +4%, tie, A +3%, A +1%.

Data set 2: A +2%, A +4%, tie, A +3%, B +4%.

The difference is that in the second case one pollster is shifted by 5% toward candidate B, approximately corresponding to the Rasmussen effect. This single outlier poll brings the average margin toward candidate B, and increases the uncertainty considerably:

Data set 1 (averages): Candidate A leads by 2.0 ± 0.7 % (mean ± SEM), win probability 98%.

Data set 2 (averages): Candidate A leads by 1.0 ± 1.4%, win probability 74%.

However, now use medians. The two data sets have the same median, 2.0%. Median-based statistics allow calculation of estimated SD, defined as (median absolute deviation)*1.4826. This gives

Data set 1 (medians): Candidate A leads by 2.0 ± 0.7% (median ± estimated SEM), win probability 98%.

Data set 2 (medians): Candidate A leads by 2.0 ± 1.3%, win probability 90%.

Generally speaking, using medians gets rid of most of the bias from a single outlier. In this example, the race is taken most of the way out of the toss-up category.

You might ask: if medians are so great, then why don't popular aggregators like FiveThirtyEight use them? A big one is that media organizations want to maintain an appearance of neutrality. I argue that a simple tool, the median, solves the problem, improves the quality of aggregated data, and helps cut through the noise -- which is why we like poll aggregation in the first place.

EMAIL TO A FRIEND X
Your Email has been sent.
You must add at least one tag to this diary before publishing it.

Add keywords that describe this diary. Separate multiple keywords with commas.
Tagging tips - Search For Tags - Browse For Tags

?

More Tagging tips:

A tag is a way to search for this diary. If someone is searching for "Barack Obama," is this a diary they'd be trying to find?

Use a person's full name, without any title. Senator Obama may become President Obama, and Michelle Obama might run for office.

If your diary covers an election or elected official, use election tags, which are generally the state abbreviation followed by the office. CA-01 is the first district House seat. CA-Sen covers both senate races. NY-GOV covers the New York governor's race.

Tags do not compound: that is, "education reform" is a completely different tag from "education". A tag like "reform" alone is probably not meaningful.

Consider if one or more of these tags fits your diary: Civil Rights, Community, Congress, Culture, Economy, Education, Elections, Energy, Environment, Health Care, International, Labor, Law, Media, Meta, National Security, Science, Transportation, or White House. If your diary is specific to a state, consider adding the state (California, Texas, etc). Keep in mind, though, that there are many wonderful and important diaries that don't fit in any of these tags. Don't worry if yours doesn't.

You can add a private note to this diary when hotlisting it:
Are you sure you want to remove this diary from your hotlist?
Are you sure you want to remove your recommendation? You can only recommend a diary once, so you will not be able to re-recommend it afterwards.
Rescue this diary, and add a note:
Are you sure you want to remove this diary from Rescue?
Choose where to republish this diary. The diary will be added to the queue for that group. Publish it from the queue to make it appear.

You must be a member of a group to use this feature.

Add a quick update to your diary without changing the diary itself:
Are you sure you want to remove this diary?
(The diary will be removed from the site and returned to your drafts for further editing.)
(The diary will be removed.)
Are you sure you want to save these changes to the published diary?

Comment Preferences

  •  538 says it already adjusts for house effect (0+ / 0-)

    The diary states:

    You might ask: if medians are so great, then why don't popular aggregators like FiveThirtyEight use them? A big one is that media organizations want to maintain an appearance of neutrality.
    538 says it adjusts for house effect.
    http://fivethirtyeight.blogs.nytimes.com/...

    After the weighted polling average is calculated, it is subject to three additional types of adjustments.

    . . .

    The house effects adjustment. Sometimes, polls from a particular polling firm tend consistently to be more favorable toward one or the other political party. Polls from the firm Rasmussen Reports, for example, have shown results that are about 2 points more favorable to the Republican candidate than average during this election cycle. It is not necessarily correct to equate a house effect with “bias” – there have been certain past elections in which pollsters with large house effects proved to be more accurate than pollsters without them – and systematic differences in polling may result from a whole host of methodological factors unrelated to political bias. This nevertheless may be quite useful to account for: Rasmussen showing a Republican with a 1-point lead in a particular state might be equivalent to a Democratic-leaning pollster showing a 4-point lead for the Democrat in the same state. The procedure for calculating the house effects adjustment is described in more detail here. A key aspect of the house effects adjustment is that a firm is not rewarded by the model simply because it happens to produce more polling than others; the adjustment is calibrated based on what the highest-quality polling firms are saying about the race.

  •  The other issue is that Ras will often poll more (0+ / 0-)

    often than other pollster, so they would shift even the medium numbers.

    -1.63/ -1.49 "Speaking truth to power" (with snark of course)!

    by dopper0189 on Tue Aug 07, 2012 at 02:01:04 PM PDT

Subscribe or Donate to support Daily Kos.

Click here for the mobile view of the site