Skip to main content

So, I've been a fair amount of work on how to weight polls from different pollsters and from different periods of time.  It beats what I consider to the the depressing tenor of the Democratic campaign in recent days, and it's part of a project I'm working on to build some prediction models for the general election, which I hope will turn into an off-site project with frequent updates to Daily Kos.  You ever wanted a version of the RCP averages that is smart enough to weight a Survey USA poll more than an ARG poll?  That's what I'm going to try and do.

In any event, Survey USA has been publishing pollster report cards that assess the average error between different pollsters in all the primary races to date.  I have one significant critique of Survey USA's methodology, which is that they tend to treat all primaries as being equal.  For example, there are a lot of primaries (and caucuses) where all the pollsters missed big: South Carolina and New Hampshire are two prominent examples.  And, as it so happened, Survey USA tended to sit out some of the races -- like New Hampshire, and Wisconsin -- that other pollsters missed badly.  So, I want to do a version of analysis that builds in an adjustment for "degree of difficulty".

But first, let's do a version of what Survey USA did, and simply look at the average error for each of a wide group of polling firms.  I tracked a total of 17 different polling firms; this should include all the usual suspects and then some:    

American Research Group (ARG)
CNN / Opinion Research
FOX / Opinion Dynamics
Gallup
Insider Advantage
LAT / Bloomberg
Marist
Mason-Dixon
Public Policy Poling (PPP)
Quinnipiac
Rasmussen
Research 2000
Selzer & Co
Strategic Vision
Suffolk
Survey USA
Zogby

A couple of other important notes: I only looked at races in which at least three pollsters from the above group surveyed the election within 10 days of the primary or caucus.  This includes AL, CA, CT, FL, GA, IA, IL, MA (Dem only), MD, MI (GOP only), MO, NH, NJ, NV, NY, SC, TN, VA and WI.  As I intimated, I only included a firm if its poll was released within 10 days of that state's election.  Also, one other matter of definition: I have simplified each race to a two-candidate affair.  For the Democrats, this is straightforward: I evaluate the Clinton-Obama margin.  For the Republicans, however, it depends on which two Republicans were most competitive in that race.  This is McCain-Romney in CA, CT, FL, IL, MI, NV, NH, NJ, and NY.  It is McCain-Huckabee in AL, GA, MD, MO, SC, TN, VA and WI.  And it is Huckabee-Romney in Iowa.

Here are those results:

In general, my results should be pretty similar to Survey USA's.  Survey USA really has done quite well.

However, are the same pollsters that have done well in this year's primaries -- like Survey USA -- also those that have done well in other recent elections?  Next, let's take a look at the average error for general election states in 2004.  I included 26 states, (I won't bother to list them) that at least three pollsters from our group surveyed within 15 days of the general election date, plus each pollster's final national number (I allow a slightly wider time frame because general election numbers tend to be much more stable than primary election numbers).

The first thing that jumps out is how much lower these numbers are across the board.  Surveying the general election is degrees of magnitude easier than surveying the primaries, because there are many fewer swing voters, and because turnout is far more stable from election to election (although, I suspect this year's general will have both more swing voters and less predictable turnout than the last couple of elections).

But -- the pollsters that have done well in the primaries this year are not necessarily the same ones who did well in 2004.  Rasmussen and Mason-Dixon, who have had fairly rough primary cycles, both did pretty well in the general election.  ARG did pretty well in the general election, for that matter.  Survey USA's results, on the other hand, were pretty marginal.

It may be that the pollsters who have had trouble in the primaries are exactly those who try and apply conservative, general election type turnout assumptions to the primaries.  More likely, however, there is simply a lot of luck involved, which is why it helps to look at as much data as possible.  In that spirit, let's look at battleground races for Senate (2006 and 2004) and Governor, as defined by Real Clear Politics.  There are a total of 33 contests that qualified.

This time, it's Research 2000 that comes out looking good, though Rasmussen and to a lesser extent Mason-Dixon again did fairly well for themselves.  Survey USA was among the lesser pollsters among those that surveyed a sufficient number of races.

Anyway, it's clear that we need a little bit of a finer-toothed comb to pick through these results and sort everything out.  So, what I've done below is to compare the error for each pollster to the average error of all pollsters in that election.  Why is this important?

Well, for example, PPP would at first glance look to have done badly in South Carolina.  They predicted Obama to win by 20 points, when he actually won by 29 -- so they missed by 9 points.  But the average pollster apart from PPP missed South Carolina by an average of 18 points.  PPP actually deserves some credit for getting the race relatively close.  

So, the next version of this analysis compares each pollster's average error to the average error of all pollsters in the contests that it surveyed -- in effect, that degree of difficulty adjustment that I described above.  Note that I am weighting these numbers by the number of pollsters that surveyed a given contest.  This is for a couple of reasons.  Firstly, we have more points of comparison for states and contests that were surveyed by more agencies.  It's one thing to get a race right when two other firms get it wrong, and another thing to get it right when ten other firms get it wrong.  And secondly, since the more "important" races tend to get surveyed more often, this has the effect of placing more emphasis on higher-profile races.

The results shift around a bit when we account for this degree of difficulty factor.  PPP benefits a bit, for instance, as I suggested above.  In contrast, a pollster like Quinnipiac, which tends to stick to "safe" races in big states, looks like an average pollster rather than a good one.

However, there is yet one last wrinkle I want to include.  Different pollsters tend to survey different races.  Some poll primaries heavily, but not the general election.  Some will survey a lot of contests, while others are more regionally-based, or tend only to make cameos from time to time.  What if a pollster looks good by comparison because it happens to be matched up frequently against ARG or Zogby?  Or, it looks bad because it often has to battle Survey USA?  The way I did this was to iterate the results several times, in much the same way that strength-of-schedule adjustments are calculated for the NFL or college basketball.  Although this extra degree of detail does not make all that much difference, I've shaded the resultant table in yellow to highlight the fact that this is the "official" version of my pollster ratings:

We see Insider Advantage move down a bit -- it does seem to be matched up against ARG fairly often -- but otherwise the results are fairly stable.  

For the most part, the winners and losers should be fairly self-evident.  Selzer polls very much live up to their stellar reputation -- they're a real outlier in the positive direction -- although, with just 6 polls in the database, we probably shouldn't make too much of this result.  Survey USA really is pretty good; it's awesome performance in the primaries is enough for it to end up way on the positive side, even after accounting for its marginal performance in general election matchups.  Research 2000 also ends up looking good, which is one I wouldn't have guessed; they do pick-and-choose their races a little bit, but their results have been solid.

Rasmussen and Mason-Dixon are two pollsters that survey a ton of races and perform just slightly better than average.  I'd be a little skeptical of any remaining primary polls they conduct, which tend to be based on overconservative turnout models, but for the general election they should be pretty solid.  The new kid on the block, PPP, has done comparatively pretty well so far, although with just 11 contests under its belt, it's too early to come to any conclusions.  And Quinnipiac and (surprisingly?) Strategic Vision seem to be just fine.

The biggest losers, predictably, are ARG and Zogby; note that I haven't included the Zogby Interactive polls, in which case they'd do even worse.  Zogby hasn't actually been all that terrible in the primaries -- they'd actually rate as slightly above average in the primaries alone if you throw out their big mistake in the California Democratic primary -- but their long history of poor performance warrants plenty of skepticism.  ARG, on the other hand, just seems to be getting worse and worse; their numbers have been off-the-charts bad in the primary cycle thus far.

Insider Advantage have been below average, but not completely terrible.  Suffolk and Marist don't tend to survey many races, and when they do, they aren't much help.  But, the other big group of losers are those polls associated with major news organizations.  LAT (via Bloomberg), CNN (via Opinion Research) and FOX (via Opinion Dynamics) have all underperformed.  So has Gallup, which had some bad misses (like Kerry taking Ohio and Florida by comfortable margins) in 2004.  

How much difference does this all make?  For a good pollster like Survey USA, which comes an average of 1 point closer than an average pollster, this is the equivalent of having a margin of error that is 1.25 points lesser.  And for a bad pollster like ARG, which performs about 1.25 points worse than average, this is the equivalent of having an extra 1.56 points appended to its margin of error.  The way this works out, ARG would have to have a sample size of about 1200 people to be equivalent to a Survey USA poll that samples just 300 people.  In that sense, the very best pollsters really might be several times more reliable than the very worst ones.

Originally posted to poblano on Mon Mar 03, 2008 at 10:17 PM PST.

EMAIL TO A FRIEND X
Your Email has been sent.
You must add at least one tag to this diary before publishing it.

Add keywords that describe this diary. Separate multiple keywords with commas.
Tagging tips - Search For Tags - Browse For Tags

?

More Tagging tips:

A tag is a way to search for this diary. If someone is searching for "Barack Obama," is this a diary they'd be trying to find?

Use a person's full name, without any title. Senator Obama may become President Obama, and Michelle Obama might run for office.

If your diary covers an election or elected official, use election tags, which are generally the state abbreviation followed by the office. CA-01 is the first district House seat. CA-Sen covers both senate races. NY-GOV covers the New York governor's race.

Tags do not compound: that is, "education reform" is a completely different tag from "education". A tag like "reform" alone is probably not meaningful.

Consider if one or more of these tags fits your diary: Civil Rights, Community, Congress, Culture, Economy, Education, Elections, Energy, Environment, Health Care, International, Labor, Law, Media, Meta, National Security, Science, Transportation, or White House. If your diary is specific to a state, consider adding the state (California, Texas, etc). Keep in mind, though, that there are many wonderful and important diaries that don't fit in any of these tags. Don't worry if yours doesn't.

You can add a private note to this diary when hotlisting it:
Are you sure you want to remove this diary from your hotlist?
Are you sure you want to remove your recommendation? You can only recommend a diary once, so you will not be able to re-recommend it afterwards.
Rescue this diary, and add a note:
Are you sure you want to remove this diary from Rescue?
Choose where to republish this diary. The diary will be added to the queue for that group. Publish it from the queue to make it appear.

You must be a member of a group to use this feature.

Add a quick update to your diary without changing the diary itself:
Are you sure you want to remove this diary?
(The diary will be removed from the site and returned to your drafts for further editing.)
(The diary will be removed.)
Are you sure you want to save these changes to the published diary?

Comment Preferences

  •  tip jar (122+ / 0-)

    Barack Obama. Because we can do better.

    by poblano on Mon Mar 03, 2008 at 10:16:30 PM PST

  •  Great work . . . (6+ / 0-)

    and useful diary as usual.  Recommended.

  •  i almost wish there was a diary hall of fame (15+ / 0-)

    or something where some diaries that contain accepted and important information could be viewed by any newcomers. something like who the most reliable pollsters are is certainly something of value to everyone.

    The only limit to our realization of tomorrow will be our doubts of today. -Franklin Roosevelt, US President 1932-1945

    by Liberal Youth on Mon Mar 03, 2008 at 10:26:07 PM PST

  •  Excellent analysis (4+ / 0-)
    Recommended by:
    Delilah, ivote2004, SherriG, Femlaw
  •  Wow (6+ / 0-)

    Fantastic work! It's amazing how much trouble previously good pollsters are having with these primaries. ARG, Mason Dixon, and Rasmussen, in particular, have really dropped off a cliff. Zogby was actually one of the best pollsters way back when, but have sucked for years now.

  •  Thanks for your effort. (2+ / 0-)
    Recommended by:
    ivote2004, la urracca

    Big respect for this in-depth analysis.

  •  outstanding work... if you have a chance ... (3+ / 0-)
    Recommended by:
    ivote2004, elmo, la urracca

    can you give some feedback on Al's projection on Texas delegates over at The Field?
    slightly off topic, i know ... but just wondering...

    •  I second that motion. (0+ / 0-)

      Thanks, Boatsie for that great link to The Field.

      For the first time, I have an analytical model for why it feels the way it does in El Paso ;)

      #3: defend network neutrality; #2: defend electoral integrity; #1: defend ecosystemic sustainability.

      by ivote2004 on Tue Mar 04, 2008 at 01:08:54 AM PST

      [ Parent ]

    •  What do quant folks here think of the analysis (0+ / 0-)

      in Giordano's latest post:

      You heard it here, first: Clinton’s "3 a.m. ad" has backfired in the Lone Star state.

      On power-rotation on Texas TV, Mark Penn’s wet dream of finally getting to go full-powered nasty against Obama is not only the laughing stock of YouTube but also among most young Texans, including young Hispanic Texan voters, and no small number of elder ones, too.

      The opposite of what Penn intended has happened: the ad has pushed white male voters toward Obama, and will cause a higher-than expected Independent and Republican cross-over to vote against Clinton in the Democratic primary, tossing various delegates from rural, white, Republican districts into the Obama column.

      Ironically, Rush Limbaugh’s call for Republicans to cross-over and vote for Clinton to "bloody up Obama" will likewise have unintended consequences: planting the idea – in a sense, "giving permission" to Republicans to cross party lines – but to people who never in a million years would vote for a Clinton, and who generally like Obama as a person, if not for the tone of his politics.

      If you think that Clinton’s "3 a.m." ad has worked for her, consult today’s SurveyUSA poll, taken after the ‘3 a.m.’ ad hit the airwaves: Among Texan voters that consider the Iraq war the most important issue, 57 percent favor Obama to 42 for Clinton. Among those that see "terrorism" as the most important issue, 62 percent back Obama to just 30 for Clinton. The survey samples are small on that – caveat emptor – but they also reflect the word on the street.

      Or if you missed it earlier, check out this video of real-time focus group results of three kinds of people – Clinton supporters, Obama supporters, and undecided voters - watching that ad.

      Remember that the "3 a.m." ad never ran in Ohio. Too bad for Obama. Because what it has wrought in Texas is a likely delegate blowout tomorrow on his behalf.

      With just 53 percent of the vote – my conservative estimate – Obama will pick up a net gain of 16 delegates from the daytime primary vote, and, at minimum, a net plus of 11 delegates in the evening caucuses (although that is a conservative estimate), for a total advance of 27 delegates in Texas on March 4.

      Combined with The Field estimates for Ohio, Rhode Island and Vermont, that will give Obama a net gain of 20 pledged delegates nationwide, bringing his grand total to 182 and then watch the superdelegate floodgates open.

      #3: defend network neutrality; #2: defend electoral integrity; #1: defend ecosystemic sustainability.

      by ivote2004 on Tue Mar 04, 2008 at 01:15:17 AM PST

      [ Parent ]

    •  Poblano, I second that... (0+ / 0-)

      Are you going to give us a Mini-Super Tuesday prediction?

      Thanks for all of your work, it is always quite informative.

      Go Obama & Go Dems,

  •  Well done, poblano (1+ / 0-)
    Recommended by:
    la urracca

    Thanks for your efforts.

    :-)

  •  Now, if you hd included the price schedules... (0+ / 0-)

    you'd have a diary worthy of Consumer Reports.

    Nice work there.

    Freedom, like responsibility, is a choice you have to make everyday.

    by papicek on Mon Mar 03, 2008 at 11:09:31 PM PST

  •  Lots of number crunching (0+ / 0-)

    to find out that polls are generally wrong.  

    Which we all knew.

    Which is why we actually vote.

    The people who are involved in political campaigns need to have some idea about where their candidates stand, and polls are the only "scientific" way to get an idea about that.  The problem is that the rest of us (and the news media) use them for entertainment, pulling "narratives" out of our/their asses, and forming opinions based on these suspect numbers, which, ultimately, mean nothing.

    Your research is valuable and fun to read Poblano, but I wish "lay people" would stop being so obsessed with polls, and instead turn their energy to discussing the merits of actual issues.  

  •  Currently (1+ / 0-)
    Recommended by:
    ivote2004

    this is the only reliable poll that I know of:
    Bush Polls

  •  Transparency should be a factor (0+ / 0-)

    Since I'm not on the ground, I read a lot of these polls. What kills me is the fact that many pollsters hide a lot of their internals. As we've seen this season, a good sampling is essential to a reliable poll. Variables such as age, gender, ethnic origin, education, income have influenced my perception of different polls.

    Problem is, many pollster hide their methodology behind a paywall or don't publish them. Sometimes, you don't even get the sample size or the margin of error. I'd like to see a more uniform way of publishing polls and I'd like to see more prominent warnings when pollsters or journalist use crosstabs to make an analysis. After all, a subsample of 80 people breaking 50-50 could break heavily either way.

  •  I think (3+ / 0-)
    Recommended by:
    smintheus, HeyMikey, rcbowman

    you need to factor in how far out from the election each poll is, especially in the primaries, since there is often real movement in the last 5-8 days. If some of these pollsters consistently have their last poll at different times than others, that could substantially skew the results. (I think this would make Zogby look even worse.)

    •  exactly (0+ / 0-)

      This is potentially a very large factor. Most states have seen campaigning in earnest only within the final 10 days before the primary, and in many, a large initial Clinton lead dwindled or evaporated. So really, it's only fair to compare polls taken on the eve of the vote against the vote itself. Polls are not meant to be predictive of outcomes; they're a snapshot of where the race stands during the period of the poll.

  •  Double wow. Great research, clear ... (1+ / 0-)
    Recommended by:
    taylormattd

    ...writing, compelling analysis. Iron my shirt!!

    Just kidding about that last. Multiple kudos on this Diary, poblano. Again.

    The limits of tyrants are prescribed by the endurance of those whom they oppose. - Frederick Douglass

    by Meteor Blades on Mon Mar 03, 2008 at 11:41:45 PM PST

  •  Fantastic stuff (0+ / 0-)

    But I think there is more at work this year than just the usual primary/general difference.

  •  what about getting the winner/ranking right? (0+ / 0-)

    Could someone just clarify if getting the winner right counts for something extra weight, even if you deviate further in the numbers for each candidate?

    If I understand the methodology correctedly (at the website of SUSA's polling report card), pollster A can have a smaller "error" than pollster B despite getting the winner wrong while pollster B gets this right.  The website says:

    Error is calculated using the "Mosteller 5″ Measure, which works this way: If "Pollster A" says Smith beats Jones by 8 points, and in fact Smith beats Jones by 5 points, then Pollster A has an "error" of 3 [8 minus 5]. A 3 would be recorded in the table above for that contest.

    So say pollster A says it's going to be Clinton 50, Obama 45, pollster B says Obama  48, Clinton 47, but actual result is Clinton 48, Obama 47, does pollster B have the lower "error" despite getting the winner wrong?  Just wondering.  Maybe, with these statistical stuff, you shouldn't deserve any extra points for at least getting the winner right.  Am just an ignorant layperson wondering about this

    •  Getting the winner right (1+ / 0-)
      Recommended by:
      smintheus

      is the wrong way to measure error, and has led to a lot of the confusion about prior polls.  If an election is very close (or, in a GE presidential race, close in some states) then getting the winner right is almost like a coin toss.

      Say the following happens in a race:

      Pollster A says  Obama wins NM by 10 points
      Pollster B says  McCain wins NM by 1 point

      NM goes for Obama by 1 point.

      Which poll was better?  The second.

  •  Primary election poll predictions are (1+ / 0-)
    Recommended by:
    smintheus

    more difficult than general election predictions for at least four reasons.

    I have a few suggestions for the next version of your research.

    (1) Your analysis combines predictions of both primaries and caucuses.  My hunch is that the "turnout" or mobilization effects are significantly stronger in caucus states than in primary states.  You could confirm (and possibly adjust for whether the average error is higher in caucuses. (Using a regression approach, you could put in a dummy variable for whether the state was a caucus state.) The expertise needed to predict the effect of the GOTV efforts in caucus states is greater than that needed to predict the effects of such efforts in primary states, where a variety of "traditional" assumptions might work.

    (2) Your analysis combines predictions from multi-candidate races with those from essentially two-candidate races.  My hunch is that pollsters do much better in predicting two-horse races than in larger fields (which might also imply that they do better later in the primary season than earlier, as well as better in general elections than primaries).  You could assess this directly, and, if reasonable you could adjust your error assessment for the number of competitive candidates (esp. if these aren't just token candidates who get just 1 to 4 percentage points).

    (3) In interpreting differences in accuracy between the primaries and general elections, in addition to the effects mentioned above (caucus vs. primary, number of competitive candidates) you might consider that in general elections the pollsters have more baseline information to work from.  Specifically, in general elections the pollsters have the partisan division of previous races, and of previous polls, to work with.  This means that in a two-party partisan race, they have an expected outcome from previous surveys and elections.  

    They can, and often do, use estimates of turnout and of partisan preferences from previous polls and elections, perhaps within regions or segments of the population (race, ethnicity) within the state to weight the raw poll results.  Using such weights helps to overcome differential nonparticipation or nonresponse by members of different partisan or racial-ethnic groups.  If their reported results are adjusted for party ID, they are likely to be more accurate. (This information is often but not always reported by pollsters. You may have to dig for their methodology reports concerning how they weight the data -- simply by demography, or also by partisanship.)

    (4) The number of DK's affects the accuracy.  Whether you were to use the "Keith number" or some other indicator (since you have also pointed out in previous diaries that pollsters differ in how hard they probe for a preference), you might try to take into account how "closed" or completed the race is by measuring the reported percentage of undecideds.

    My guess is that you have, or can get, information to take all of the above factors into account.  

    Another to consider is something that has been remarked on elsewhere:  the tendency of African American respondents in particular to underreport their "true" preferences, i.e., to overreport themselves as "undecided." (Why this is so is worthy of analysis, but think one reason may be a certain distrust of anonymous pollsters owing to the past history of vote suppression against African Americans.) I would surmise that in partisan elections, you are going to see much larger "Keith numbers" among African Americans than among whites.  An experienced pollster might adjust the data accordingly, and/or realize that if a large percentage of the electorate in a state are African Americans the overall poll results are probably going to underestimate the eventual Democratic proportion of the vote.

    •  regarding the two-candidate vs. multi-candidate (0+ / 0-)

      one way to do this is to just look at accuracy of share of the person who won the election.

      Or, if one really wanted to go bananas, one could simulate each race, using the final vote as 'truth' and see how often a result as far off as each pollster's would come up.  That would give you a measure of accuracy that was unaffected by margin, or by number of viable candidates

  •  fantastic diary (0+ / 0-)

    with the reality check of Selzer doing well, ARG not.

    I think you're on to something.

    "Politics is the art of looking for trouble, finding it everywhere, diagnosing it incorrectly and applying the wrong remedies." - Groucho Marx

    by Greg Dworkin on Tue Mar 04, 2008 at 03:45:01 AM PST

  •  Poblano (0+ / 0-)

    This is somewhat off topic, but I am hoping I can get you to respond to a burning question that I have this morning. I have glanced at the futures markets which as you have pointed out before only give a sense of the common wisdom on where the races stand. You also observed that our common wisdom is probably at least as good as theirs, and I agree with that. But what I would like is for you to share your quite uncommon wisdom about how you think Ohio and Texas will turn out today. If you have a sense about who is likely to win what and by how much and would be willing to share it, it would  be a big help to me in managing my anxieties for the day. So there it is - a huge, totally sincere compliment - that I offer in the hopes of getting your predictions. Thanks  

    "A republic, if you can keep it." Benjamin Franklin

    by herodotus on Tue Mar 04, 2008 at 04:44:28 AM PST

  •  Your GE model (0+ / 0-)

    You should use this in your general election model.

    When you run your OLS regression on the polling, weigh it by poll accuracy.

  •  Thanks, poblano (0+ / 0-)

    You nourish my inner geek!

    "You can't depend on your judgement when your imagination is out of focus."
    . . . . . . . . . Mark Twain

    by Land of Enchantment on Tue Mar 04, 2008 at 10:51:16 AM PST

Subscribe or Donate to support Daily Kos.

Click here for the mobile view of the site