Skip to main content

One of the sites we've come to rely on for objective data on the ups and downs of campaigns and public opinion is the website — great graphics if you haven't visited, and unique analysis by Mark Blumenthal (aka the Mystery Pollster) and Prof. Charles Franklin (he of the regression analysis curves), and guest pollsters from all over the political map. But an analysis site like this one is only as good as the data that is available to analyze, and that data is sometimes frustratingly incomplete. Anyone looking for a poll's crosstabs that don't seem to exist will appreciate the following post from Mark Blumenthal:

Why do so many pollsters disclose so little? A few continue to cite proprietary interests. Some release their data solely through their media sponsors, which in the past limited the space or airtime available for methodological details (limits now largely moot given the Internet sites now maintained by virtually all media outlets and pollsters). And while none say so publicly, my sense is that many withhold these details to avoid the nit-picking and second guessing that inevitably comes from unhappy partisans hoping to discredit the results.

Do pollsters have an ethical obligation to report methodological details about who they sampled? Absolutely (and more on that below), and as we have learned, most will disclose these details on request as per the ethical codes of the American Association for Public Opinion Research (AAPOR) and the National Council on Public Polls (NCPP). Regular readers will know that we have received prompt replies from many pollsters in response to such requests (some pertinent examples here, here, here and here).

The problem with my occasional ad hoc requests is that they arbitrarily single out particular pollsters, holding their work up to scrutiny (and potential criticism) while letting others off the hook. My post a few weeks back, for example, focused on results from Iowa polls conducted by the American Research Group (ARG) that seemed contrary to other polls. Yet as one alert reader commented, I made no mention of a recent Zogby poll with results consistent with ARG. And while tempting, speculating about details withheld from public view (as I did, incorrectly, in the first ARG post), is even less fair to the pollsters and our readers.

So I have come to this conclusion: Starting today we will begin to formally request answers to a limited but fundamental set of methodological questions for every public poll asking about the primary election released in, for now, a limited set of states: Iowa, New Hampshire, South Carolina or for the nation as a whole. We are starting today with requests emailed to the Iowa pollsters and will work our way through the other early states and national polls over the next few weeks, expanding to other states as our time and resources allow.[bolded mine]

More on the goal, and a request from Blumenthal:

Our goal is to both collect this information and post it alongside the survey results on our poll summary pages, as a regular ongoing feature of Obviously, some pollsters may choose to ignore some or all of our requests, but if they do our summary table will show it. We are starting with Iowa, followed by New Hampshire, South Carolina and the national surveys, in order to keep this task manageable and to determine the feasibility of making such requests for every survey we track...

What can you do? Frankly, we would appreciate your support. If you have a blog, please post something about the Pollster Disclosure Project and link back to this entry (and if you do, please send us an email so we can keep a list of supportive blogs). If not, we would appreciate supportive comments below. And of course, criticism or suggestions on what we might do differently are also always welcome.

The specific questions about demographics and methodology are below the fold.

The bottom line is that if you want better data to analyze, then we, the consumers of all things political, ought to support in asking for it. And if we expect and appreciate the analysis done by, Swing State Project, Open Left, Slate, Real Clear Politics or any of the other sites that digest and analyze polling data, let's help make the data a bit more "open source" and transparent.

These are the questions:

  • Describe the questions or procedures used to select or define likely voters orlikely caucus goers (essentially the same questions I asked of pollsters just before the 2004 general election).
  • The question that, as Gary Langer of ABC News puts it, "anyone producing a poll of 'likely voters' should be prepared to answer:" What share of the voting-age population do they represent? (The specific information will vary from poll to poll; more details on that below).
  • We will ask pollsters to provide the results to demographic questions and key attributes measures among the likely primary voter samples. In other words, what is the composition of each primary voter sample (or subgroup) in terms of gender, age, race, etc.?
  • What was the sample frame (random digit dial, registered voter list, listed telephone directory, etc)? Did the sample frame include or exclude cell phones?
  • What was the mode of interview (telephone using live interviewers, telephone using an automated, interactive voice response [IVR] methodology, in-person, Internet, mail-in)?
  • And in the few instances where pollsters do not already provide it, what was the verbatim text of the trial heat vote question or questions?

Originally posted to Daily Kos on Tue Sep 25, 2007 at 02:58 AM PDT.

Your Email has been sent.
You must add at least one tag to this diary before publishing it.

Add keywords that describe this diary. Separate multiple keywords with commas.
Tagging tips - Search For Tags - Browse For Tags


More Tagging tips:

A tag is a way to search for this diary. If someone is searching for "Barack Obama," is this a diary they'd be trying to find?

Use a person's full name, without any title. Senator Obama may become President Obama, and Michelle Obama might run for office.

If your diary covers an election or elected official, use election tags, which are generally the state abbreviation followed by the office. CA-01 is the first district House seat. CA-Sen covers both senate races. NY-GOV covers the New York governor's race.

Tags do not compound: that is, "education reform" is a completely different tag from "education". A tag like "reform" alone is probably not meaningful.

Consider if one or more of these tags fits your diary: Civil Rights, Community, Congress, Culture, Economy, Education, Elections, Energy, Environment, Health Care, International, Labor, Law, Media, Meta, National Security, Science, Transportation, or White House. If your diary is specific to a state, consider adding the state (California, Texas, etc). Keep in mind, though, that there are many wonderful and important diaries that don't fit in any of these tags. Don't worry if yours doesn't.

You can add a private note to this diary when hotlisting it:
Are you sure you want to remove this diary from your hotlist?
Are you sure you want to remove your recommendation? You can only recommend a diary once, so you will not be able to re-recommend it afterwards.
Rescue this diary, and add a note:
Are you sure you want to remove this diary from Rescue?
Choose where to republish this diary. The diary will be added to the queue for that group. Publish it from the queue to make it appear.

You must be a member of a group to use this feature.

Add a quick update to your diary without changing the diary itself:
Are you sure you want to remove this diary?
(The diary will be removed from the site and returned to your drafts for further editing.)
(The diary will be removed.)
Are you sure you want to save these changes to the published diary?

Comment Preferences

  •  Polls. (2+ / 0-)
    Recommended by:
    musicsleuth, danmac

    I'm reading Prof. Stuart Ewan's 'PR! A Social History of Spin' and it seems that opinion polls and research have been essential for decades....all the better to control us with.

    We don't take charge of our own democracy and economy......Instead, those who are at the helm get the information....then they spin it back at us......all the better to sell us on ideas, sell us products, change our habits and beliefs.

    I do LOVE full disclosure and transparency, though.

  •  Great idea. But, I just don't see the polling (0+ / 0-)

    firms answereing the questions.

  •  Pollsterdotcom is a great site (1+ / 0-)
    Recommended by:

    as is Charles Franklin's own site political arithmetik

    •  Yes (2+ / 0-)
      Recommended by:
      lgmcp, plf515

      political arithmetik was my first love in the polling sites.
      A fun, though of course very biased, site is pollkatz.  He does a great graphic of flushing Bush (using poll numbers from the years of the Bush presidency).  Also an interesting graphic of gas prices aligned with Bush's poll numbers.  It doesn't pretend to be a neutral site.  He's like the Michael Moore of polling.

      If, in our efforts to win, we become as dishonest as our opponents on the right, we don't deserve to triumph.

      by Tamar on Tue Sep 25, 2007 at 06:55:14 AM PDT

      [ Parent ]

  •  Has anyone read The Averaged American ? (0+ / 0-)

    I've seen mixed reviews, but have not read the book.  I probably should.

  •  Polling the pollsters. Soon, they (0+ / 0-)

    will be polling you.  What's next, the Escher Poll?

    Because everyone has one. Having credibility when making an argument is the straightest path to persuasion.

    by SpamNunn on Tue Sep 25, 2007 at 05:09:53 AM PDT

  •  Divination (1+ / 0-)
    Recommended by:

    They probably don't disclose their methods for the same reason mediums and psychics don't--it's largely based on pseudoscience & it would be embarrassing to disclose exactly how the raw results are tweaked. Even the base  criterion of how a poll is designated as "scientific"--the random sample--has been seriously undermined by cell phones, call-screening, VoIP and other sampling biases.

    •  see articles here (1+ / 0-)
      Recommended by:

      AAPOR and other professional societies are tackling those issues, thorny though they may be.

      "Politics is the art of looking for trouble, finding it everywhere, diagnosing it incorrectly and applying the wrong remedies." - Groucho Marx

      by Greg Dworkin on Tue Sep 25, 2007 at 05:24:31 AM PDT

      [ Parent ]

    •  They talk about these issues (1+ / 0-)
      Recommended by:

      at  The cell phone issue is a big one and is being taken very seriously by good polling outfits.  There was a big article on this at when the results of the SLAITS survey showed the large number of people who have no landlines.
       I don't agree that all polling is "pseudoscience."  Some polling organizations, e.g., Harris, are incredibly careful and have even debunked claims made by other polling companies.
      The main problem with the polls is the way questions are worded.  It's very important to read the question before judging the result.  For instance, the polls I've seen that have "shown" a large portion of people believe in creationism over evolution had no option for people to respond that they believed in both the bible and evolution and didn't really see a conflict between them.  That group would probably far outnumber the people who believe evolution is blasphemy.

      If, in our efforts to win, we become as dishonest as our opponents on the right, we don't deserve to triumph.

      by Tamar on Tue Sep 25, 2007 at 07:04:15 AM PDT

      [ Parent ]

      •  My favorite example of wording bias (1+ / 0-)
        Recommended by:

        is the perennial question "should homosexual relations between adults be legal?" The problem is that older people are going to read this as "should gay sex be legal?" while younger people are going to read it as "should same-sex relationships be legally recognized?" I doubt most people under 40 would recognize "relations" as a euphemism for "sex acts" (at least in my area, it was already pretty quaint when I was a teenager in the 1970s). Yet journos always treat the responses as indicating favor/disfavor toward sodomy laws.

        Similarly, "do you support universal health care?" is a really bad question: some people will interpret "universal health care" as a totally public system (e.g. the VA or the British NHS); others will interpret it as a public payer/private provider system (e.g. Medicare or the Canadian system); still others (probably most of us here, but fewer outside the realm of political junkies) will think of it as encompassing either possibility. You simply can't assume that the results are going to reflect the last interpretation; the question really needs to be broken down into two separate ones.

        I do like conducting hearings in an actual hearing room -- John Conyers

        by ebohlman on Tue Sep 25, 2007 at 10:16:38 AM PDT

        [ Parent ]

        •  Totally agree (0+ / 0-)

          Do you remember when Clinton was accused of lying for saying he didn't "have sex" with Monica Lewinsky.  Journal of the American Medical Association published a study, conducted well before the Clinton-Lewinsky stuff, showing that the majority of college students thought "having sex" meant intercourse.  Anything less than intercourse was not "having sex."  
          I work for an organization working to expand health coverage for children.  We commissioned a poll to get at people's views and we never said "universal health care."  We were much more specific.
          However, I find it interesting that even in the polls that use that phrase, national health coverage gets a fairly high degree of support these days.  Even when the polls suggest raising taxes to pay for it!
          I think if instead of "government-run" or "universal," the question were asked "would you like to be able to go to any doctor you want with the payment administered by the government," that would get away from the notion of "government-run" or socialized medicine.  
          Most people are not aware that single-payer doesn't mean you have to go to the doctor the government chooses.  In fact, single-payer systems could offer people far more choice in providers than they now have within their profit-making insurance plans (though, as you said, it of course depends on the detail of the plan)

          If, in our efforts to win, we become as dishonest as our opponents on the right, we don't deserve to triumph.

          by Tamar on Tue Sep 25, 2007 at 11:49:27 AM PDT

          [ Parent ]

  •  As someone who spends time looking at polls (2+ / 0-)
    Recommended by:
    Garrett, 4jkb4ia

    for example, how New Hampshire changes National Polls in the Democratic Primary (it causes on average a 33 point swing between the front runner and the main challenger),  there are two instances where methodology interests me greatly:

    1.  How are they identifying likely voters in Iowa? Exit polling shows about 65% of Caucus participants are first time participants, so determining likely attendance is difficult.
    1.  What is the percentage of independants in New Hampshire polling.  Exit polling says about 40% of voters in the Dem primary will be independents.  Frequently I see far smaller percentages in polls (ARG eg).

    As an aside one of the frustrating things is how difficult it is to access old polling information.  The Roper center has a website that has much of it, but it is expensive to access information that should belong to the public.

  •  The best of social science (2+ / 0-)
    Recommended by:
    DemFromCT, lgmcp

    As a social scientist, I applaud Blumenthal.  He truly demonstrates the best in social science.  Not only is he careful, not only does he try to be objective, but also he admits when he has made a mistake.  I go to all the time and find the numbers useful and the discussions enlightening.
    Compare this thoughtful self-criticism to the awful report issued a couple of months ago by the Urban Institute, paid for by the Bush Admin, which estimated the number of uninsured children at only half that of numerous other surveys (which amazingly coincided with Bush's threat to veto any expansion of the Children's Health Insurance Program).
    Yes, they described their methodology, which is laudable since in the past they have often claimed it's proprietary; but there was no self-reflection, no questioning of their basic assumptions, no critical thinking applied to their own work.  
    Their whole report was based on supposition piled on supposition, and the methods (and therefore results) were so suspect that even other researchers in their own institute were highly critical.  

    If, in our efforts to win, we become as dishonest as our opponents on the right, we don't deserve to triumph.

    by Tamar on Tue Sep 25, 2007 at 06:51:10 AM PDT

  •  Now to beat MSM reporters over the head ... (0+ / 0-)

    With the most basic facts of polling.

    I'm thinking, of course, of the Celinda Lake thing. You didn't need crosstabs to see the flaws of that poll, which didn't keep the WaPo from running it with a completely straight face.

    The best fortress is to be found in the love of the people - Niccolo Machiavelli

    by al Fubar on Tue Sep 25, 2007 at 06:57:55 AM PDT

    •  which poll was that? (0+ / 0-)

      Do you have a link?  Sounds like the kind of thing I want to read even though it will raise my blood pressure.

      If, in our efforts to win, we become as dishonest as our opponents on the right, we don't deserve to triumph.

      by Tamar on Tue Sep 25, 2007 at 07:06:19 AM PDT

      [ Parent ]

      •  here (2+ / 0-)
        Recommended by:
        al Fubar, lgmcp

        Ask who Lake is working for. okay, it's Biden.

        CLINTON: Not A Drag
        A 9/22 Washington Poststory on a 7/21-8/5 poll by Joe Biden pollster Celinda Lake drew wide netroots condemnation. Post authors Chris Cillizza and Shailagh Murray claimed the poll showed Hillary Clinton and Barack Obama trailing Rudy Giuliani in 31 swing districts and "potentially serving as a drag on those lawmakers' reelection chances."

        Open Left's Chris Bowers did attack the story's authors, but did reassure readers "Clinton Would Not Hurt Democrats Down Ballot." From Bowers: "[T]his poll shows the exact same 6% edge for freshman Democrats in the named generic ballot no matter if the local nominee is tied to Clinton or Obama. I think the key here is that the freshman Democrats are weakened not by the attack that connects them to any specific Democrat, but rather by the attack that they will be a tool of someone outside the district. In other words, it isn't tying Democratic candidates to Hillary Clinton that hurts them, but tying them to anyone from outside the district who is perceived as having different values."

        Others were less kind. Matthew Yglesias blogs: "So Joe Biden's pollster, Celinda Lake, did some push polling about Hillary Clinton and Barack Obama, then leaked it to The Washington Post which produced this story ... why did Lake do this? ... Under the circumstances, her willingness to engage in dirty pool against the front-runners is remarkable."

        Atrios adds: "So why is "Democratic pollster" Celinda Lake running push polls about Obama and Clinton? Why are Cillizza and the Devil running them as news? More importantly, why isn't Cillizza telling us that Celinda Lake works for the rival Biden campaign."

        "Politics is the art of looking for trouble, finding it everywhere, diagnosing it incorrectly and applying the wrong remedies." - Groucho Marx

        by Greg Dworkin on Tue Sep 25, 2007 at 07:12:27 AM PDT

        [ Parent ]

  •  "Margin of error" (0+ / 0-)

    One thing I'd like to see in the reports is a sentence after "margin of error."
    Something like "In a sample of this size, 5% of the time the results in the underlying population will be higher than the margin of error and 5% of teh time, the results in the underlying population will be lower."

    •  they can just link here (0+ / 0-)

      Thus, for example, a "3 percentage point margin of error" in a national poll means that if the attempt were made to interview every adult in the nation with the same questions in the same way at the same time as the poll was taken, the poll's answers would fall within plus or minus 3 percentage points of the complete count’s results 95% of the time.

      "Politics is the art of looking for trouble, finding it everywhere, diagnosing it incorrectly and applying the wrong remedies." - Groucho Marx

      by Greg Dworkin on Tue Sep 25, 2007 at 08:11:58 AM PDT

      [ Parent ]

  •  Bravo DemFromCT and Blumenthal (2+ / 0-)
    Recommended by:
    lgmcp, danmac

    These pollsters need to publically account to the nation how they are getting their results.  When you say "Some release their data solely through their media sponsors" I immediately thought of Hillary!  Her propagandists want you to believe she's leading everywhere and she's the inevitable president!  

    These pollsters also need to understand that Americans on on to their 'tricks'.

  •  fantastic news (2+ / 0-)
    Recommended by:
    lgmcp, danmac

    the more transperency, the better we really understand what people are thinking on issues. We'll be able to better determine subtle actions that turn a poll into the "push poll" category.

    I remember a time when the American President was the leader of the free world. ****** Repeat after me: "Neoconservatism has failed America."

    by land of the free on Tue Sep 25, 2007 at 08:31:02 AM PDT

  •  Important post. (0+ / 0-)

    I'll be consulting this Blumenthal's site in preference to others, going forward!

    "The extinction of the human race will come from its inability to EMOTIONALLY comprehend the exponential function." -- Edward Teller

    by lgmcp on Tue Sep 25, 2007 at 11:34:35 AM PDT

  •  Fantastic idea (1+ / 0-)
    Recommended by:

    There is no reason that media outlets shouldn't share the full data from their opinion surveys, subject, of course, to information constraints and a modest period during which sponsoring news media have exclusive access.  So much of information is sitting unused for decades because pollsters hoard their data.

Subscribe or Donate to support Daily Kos.

Click here for the mobile view of the site