Skip to main content

I discuss possible cell phone bias on my blog, Heartless and Brainless.

Are polls misproportionally missing younger people and minorities? What are the effects? I also include a primer on polling and margin of errors. Let's dive in further below the fold.

I’ll be the first to admit that I have a cell phone fixation. I’m checking the internet on my Samsung Sync constantly. It’s my only addiction (drinking is a hobby, fantasy sports are an investment and smart, beautiful women are a calling) and I don’t apologize for it. My only regret is that I never get to take part in any polling. That got me thinking. I’m a young African-American with a college degree and no landline. That’s a prime Obama demographic and polling skips me entirely. Is there a cell phone bias? Is it larger than the debunked Bradley Effect? (If anything, there is probably a reverse Bradley Effectin swing states.)
Will this bias work out worse for McCain than it did for the 1986 Celtics? lists a Pew Research Center poll that finds landline only polls return the same results. The same study admits that it underrepresented people and minorities under 30. I’m curious how many other people like me are out there. The What Do I Know?blog discusses many of these same issues. They find that 25% of young people don’t even have landlines. Cable companies routinely threw in a digital landline as a package deal with cable and the internet. My roommates and I never even bothered to hook up the phone.

Being the amateur statistician that I am, I decided to conduct an (overly simplistic) experiment. The normal distribution curve says that all things being equal, 95% of the time, the actual results should fall within the expected results plus or minus the margin of error. Let’s look at an example. If a poll says Obama is up 2% with a 3% margin of error, it really won’t tell you who is going to win. A perfectly constructed poll could get actual results of Obama winning by 5% or losing by 1%. It can be absolutely wrong and be a flawless poll! Sometimes people say the polling was wrong even though it was within the expected margin of error. That’s what makes polling close contests so difficult. Things only get froggy when the results fall outside the expected margin of error. Think Obama vs. Hillary in New Hampshire. Even then, the best polls will be completely wrong 5% of the time. It’s only a systemic problem when the results are consistently outside the margin of error.

Now this makes perfect sense in the lab. A seemingly infinite amount of variables can invalidate the whole poll. My hypothesis is that more than 5% of the actual primary results outside the margin of error of the polls taken right before the vote. This would lead me to believe that there is an error in the way the polls were constructed. It won’t tell us exactly how though. The media also has a  bad tendency to conflate poll numbers, rendering them a lot less reliable. Let’s delve into some numbers and see what we find. If my methodology, assumptions or explanations are wrong, please discuss in the comments.

Lindsay Politics already conducted this research. This is a chart of their findings:

Friday, July 11, 2008
Polling accuracy
This is a study I did Based on the polling of the 2004, 2006 and 2008 (Primary) elections. This is the average amount of races they blow per election cycle. 0 is the perfect score.

  1. Zogby 1.67
  1. Survey USA 1.74
  1. Quinnipiac 2.00
  1. Mason-Dixon 2.13
  1. Gallup 2.33
  1. RealClearPolitics* 2.67
  1. Rassmussen 3.00
  1. Research 2000 3.00
  1. Strategic Vision 4.00
  1. ARG 7.50

*- RCP is an average of the polls. The primaries were an outlier, without them it would have a stunning 0.5

There are 100 races every presidential primary cycle, 50 Democratic and 50 Republican. We would expect the polls to completely whiff an average of five times. Almost all these polls performed far better than this. I assume the Zogby poll is their regular phone poll. Their internet poll is a lot less reliable. Another point of consideration is that was easier to predict the 2004 primaries because Kerry and Bush locked up their nominations a lot earlier. The 2006 Congressional primaries also had a lot of incumbents. I’m not sure what Lindsay Politics considers "blowing the race" but I am assuming that they mean the results fell outside the acceptable margin of error.

The polls were far more accurate than I expected. This leads me to believe that another cause will be to blame if the polls disappoint like in 2004. My theory is that likely voter models need to be tweaked. Most polls separate likely voters from registered voters. Their models indicate that people who voted in three straight elections are more likely to vote again. This obviously works against college students who aren’t old enough to vote for three cycles. The massive amount of newly registered voters leads me to believe that more young people will vote than normal. I also expect much heavier African-American turnout than normal. I predict that polls nationwide will stay open hours later to accommodate the extra demand. Expect long lines in swing states. The only problem is that no one knows how to tweak the models for this novel election.

Originally posted to Guancous on Sat Sep 13, 2008 at 01:55 PM PDT.


Has anyone contacted you for a poll?

16%36 votes
1%3 votes
1%4 votes
65%143 votes
14%31 votes

| 217 votes | Vote | Results

Your Email has been sent.
You must add at least one tag to this diary before publishing it.

Add keywords that describe this diary. Separate multiple keywords with commas.
Tagging tips - Search For Tags - Browse For Tags


More Tagging tips:

A tag is a way to search for this diary. If someone is searching for "Barack Obama," is this a diary they'd be trying to find?

Use a person's full name, without any title. Senator Obama may become President Obama, and Michelle Obama might run for office.

If your diary covers an election or elected official, use election tags, which are generally the state abbreviation followed by the office. CA-01 is the first district House seat. CA-Sen covers both senate races. NY-GOV covers the New York governor's race.

Tags do not compound: that is, "education reform" is a completely different tag from "education". A tag like "reform" alone is probably not meaningful.

Consider if one or more of these tags fits your diary: Civil Rights, Community, Congress, Culture, Economy, Education, Elections, Energy, Environment, Health Care, International, Labor, Law, Media, Meta, National Security, Science, Transportation, or White House. If your diary is specific to a state, consider adding the state (California, Texas, etc). Keep in mind, though, that there are many wonderful and important diaries that don't fit in any of these tags. Don't worry if yours doesn't.

You can add a private note to this diary when hotlisting it:
Are you sure you want to remove this diary from your hotlist?
Are you sure you want to remove your recommendation? You can only recommend a diary once, so you will not be able to re-recommend it afterwards.
Rescue this diary, and add a note:
Are you sure you want to remove this diary from Rescue?
Choose where to republish this diary. The diary will be added to the queue for that group. Publish it from the queue to make it appear.

You must be a member of a group to use this feature.

Add a quick update to your diary without changing the diary itself:
Are you sure you want to remove this diary?
(The diary will be removed from the site and returned to your drafts for further editing.)
(The diary will be removed.)
Are you sure you want to save these changes to the published diary?

Comment Preferences

  •  Tip Jar (18+ / 0-)

    Please let me know if you find any errors. I'll edit the diary. Let me know if you have any questions!

    by Guancous on Sat Sep 13, 2008 at 01:55:53 PM PDT

    •  I think even more (1+ / 0-)
      Recommended by:
      the polls miss people who are busy and perhaps more likely even to be engaged in the process. For instance, this week, in addition to being at work from 8-6 (including commuting time) I was at a political meeting of some kind three evenings and out at art openings one night. The art openings were overflowing with young people, the kind most likely to be Obama voters. They weren't home to answer their phones either, although one presumes they had their cell phones. Would they answer a call from a pollster in the middle of an event if they looked and saw it wasn't someone they knew?

      I have been called I think twice. I got messages in my answering machine of the "Are you there? Are you there?" variety. No, i wasn't. I'm TOO BUSY!

      I did answer a call once with a push poll for Ken Blackwell when he was running for governor. It included questions like "Do you think Ohio would benefit from having a strong black governor?" (They must have assumed given my address I had a fair chance of being black. I'm not). My response: can I choose the black person? I know some good candidates but they don't include Ken Blackwell!

      We're retiring Steve LaTourette (R-Family Values for You But Not for Me) and sending Judge Bill O'Neill to Congress from Ohio-14:

      by anastasia p on Sat Sep 13, 2008 at 03:10:59 PM PDT

      [ Parent ]

  •  ARG (1+ / 0-)
    Recommended by:

    called my cell phone a couple of days ago.

  •  I agree - young people weren't fired up for Kerry (2+ / 0-)
    Recommended by:
    Guancous, sociald123

    the way they are for Obama.  And that probably goes double or triple for African-American voters.  

    I'd love to find out Obama has an invisible built-in 5-point cushion in all of these polls.... not sure it's true but not sure it isn't.  I'll be interested to see if others know more.

    •  And yet, they WERE fired up for Kerry (2+ / 0-)
      Recommended by:
      Guancous, HeyMikey
      I never saw so much enthusiasm for a campaign among young people as I saw in 2004. One local rock club registered over 500 people in the lead-up to the campaign. Our campaign office in Cleveland, Ohio was flooded with young volunteers from Illinois, New Jersey and New York. But yeah, even more so this time. And the GOP is freaking out over the likelihood of fair elections and no voter suppression and is currently busy trying to promote some suppression. (I just posted a diary about their latest trick which no one is ready because I didn't put "Sarah Palin" in the tirle.)

      We're retiring Steve LaTourette (R-Family Values for You But Not for Me) and sending Judge Bill O'Neill to Congress from Ohio-14:

      by anastasia p on Sat Sep 13, 2008 at 03:13:33 PM PDT

      [ Parent ]

  •  I'm 40 - no landline (3+ / 0-)
    Recommended by:
    Guancous, blindyone, kenjisan

    "This is the twenty-first century. Nations don't just go invading other countries." --John McCain, said without a hint of irony.

    by maynard on Sat Sep 13, 2008 at 02:00:39 PM PDT

    •  55 and got rid of the landline (2+ / 0-)
      Recommended by:
      maynard, Guancous

      why have a cellphone that I take everywhere anyway, and the added cost of a landline.

      "Marry in haste, repent at leisure." Will the new saying go, ''Vote in haste, repent at leisure''? Patt Morrison

      by kenjisan on Sat Sep 13, 2008 at 02:01:54 PM PDT

      [ Parent ]

      •  exactly why I did it too (2+ / 0-)
        Recommended by:
        Guancous, kenjisan

        $50/mo for one or the other, but not both. Well, I bought a snazzy new iPhone, so that's a bit more. But still, no point in buying a landline too.

        "This is the twenty-first century. Nations don't just go invading other countries." --John McCain, said without a hint of irony.

        by maynard on Sat Sep 13, 2008 at 02:04:43 PM PDT

        [ Parent ]

  •  I', 38 -- no landline nt (1+ / 0-)
    Recommended by:

    "No way, no how, no Palin war with Russia." --KariQ

    by andrewj54 on Sat Sep 13, 2008 at 02:01:14 PM PDT

  •  I don't know anyone who has a land line (2+ / 0-)
    Recommended by:
    Guancous, blindyone

    anymore for phones, maybe for internet (my 'Publican mother), so yes, some fraction of the count is off.  Anyone who attempts to poll me while I'm driving gets put to voice mail, and I'm probably not the only one who does that.

    A people and their army who behave like this abroad invariably bring this criminal and brutish behaviour home and turn it against their own people. - Laith

    by sailmaker on Sat Sep 13, 2008 at 02:01:18 PM PDT

    •  Yeah, my children are Obama supporters (2+ / 0-)
      Recommended by:
      Guancous, kenjisan

      (in their twenties) if you don't leave a voice mail, they don't bother returning your call.

      "... he's giving you $5000. in a tax credit- that is the ultimate bridge to nowhere when it costs $12000. to get health care" Joe Biden

      by blindyone on Sat Sep 13, 2008 at 02:18:33 PM PDT

      [ Parent ]

    •  My wife and I are 46 (2+ / 0-)
      Recommended by:
      Guancous, state29

      and we own a land line as well as cell phones (OH IF ONLY MY WIFE WOULD REMEMBER TO TURN HER DAMN CELL PHONE ON WHEN SHE'S OUT!!).

      We own a land line for making local calls and for long distance calls that aren't covered by our cell phone plan.  Also, we don't feel comfortable just having cell phones in case one or both cell phones goes dead for whatever reason.  Also, the sound quality is just plain better on our land line.

  •  I'm a cell phone only HH too. when I signed up (1+ / 0-)
    Recommended by:

    for my cable package - cable, phone, internet - I NEVER plugged in the lan line.

    John McCain -- Putting the "ick" back in Maverick!!!!

    by fhamme on Sat Sep 13, 2008 at 02:02:40 PM PDT

  •  This is silly (3+ / 0-)
    Recommended by:
    slinkerwink, Guancous, Larry Bailey

    comes up every cycle.

    Early in the new year it made all the Ron Paulbots sure they would over perform.

    It's a myth that has some intuitive myth, but it doesn't play out in practice.

    We are, factually, losing the national polls right now. Don't conjecture ways that numbers might be wrong, correct the problem the rights ways: Spread the Word, Donate, Volunteer.

    •  I beg to differ. Re-read the diary... (3+ / 0-)
      Recommended by:
      Guancous, capelza, kenjisan

      cell phone HHs are under represented -- whether you like it or not!

      John McCain -- Putting the "ick" back in Maverick!!!!

      by fhamme on Sat Sep 13, 2008 at 02:05:39 PM PDT

      [ Parent ]

      •  Underrepresentation (1+ / 0-)
        Recommended by:

        My conclusion is that cell phone voters ARE being underrepresented. Even factoring this in, I found that the polls are STILL accurate. Obama is tied to slightly down right now.

        Spread the Word.

        But we should be doing that anyway! We won't know anything for sure until November 5th, if not later.

        by Guancous on Sat Sep 13, 2008 at 02:09:53 PM PDT

        [ Parent ]

  •  For now, (2+ / 0-)
    Recommended by:
    George, Guancous

    public opinion scholars agree that this is not an issue.  All credible firms use statistical adjustments to account for the underrepresentation of younger voters, minorities, etc.

    The bottom line is, if this were a problem, it would have manifest itself consistently throughout the Democratic primary, in which one candidate had an overwhelming advantage among the same demographic groups that tend toward cell only.  It didn't.

    Nobody said it would be easy.

    by cardinal on Sat Sep 13, 2008 at 02:05:02 PM PDT

  •  I don't think it's cell phones (6+ / 0-)

    it's call screening that throws them off.

    Do YOU pick up calls you don't recognize? Do YOU let them go to voicemail and then decide if it's worth calling back?

    That has nothing to do with cell or land line, it has to do with telemarketers.

    If you're like most people and screen your calls, by definition you won't be polled. And the only people who WILL be polled are those that answer calls they don't recognize. Because on almost every poll, if a machine answers THEY HANG UP. Pollsters don't leave you a voicemail. Robopollers for sure don't.

    That means, to me at least, that all phone polls are skewed to the least technologically literate, possibly uneducated, probably older, demographic. It also means they have to call a HUGE number of people to even remotely get a sample.

  •  Also :Who picks the phone up ? (2+ / 0-)
    Recommended by:
    Guancous, capelza

    Honestly if it is someone who I dont know I dont pick it up. This is especially for people who get calls from credit agencies they dont pick up.

    I think these go for Obama.

    Now it is part of any package which was not before

    "Four seconds is the longest wait " -Sleater-Kinney

    by delphil on Sat Sep 13, 2008 at 02:05:16 PM PDT

  •  Likely voters (2+ / 0-)
    Recommended by:
    Guancous, capelza

    its always been my impression that the polsters are after likely voters aka people who have voted in the last 2 elections and its my belief it doesnt cover new voters or folks who have not voted in years.. it is my optimistic belief that the cushion of 8% may be there for barack..i cant link you to these statistics its just my belief

    •  Optimistic indeed (1+ / 0-)
      Recommended by:

      If that were true, it would have showed up in the primaries, in which Obama's advantage among those groups over Clinton was nearly as big as it will likely be against McCain.  Yet, it didn't.

      We really can't count on a mythical "hidden" Obama vote.  We have to knock on enough doors and call enough people to turn it into an actual vote.

      Nobody said it would be easy.

      by cardinal on Sat Sep 13, 2008 at 02:09:45 PM PDT

      [ Parent ]

  •  Me and my husband are 25 (1+ / 0-)
    Recommended by:

    and only have cells.

  •  I think we got our answer back in Jan-March (4+ / 0-)
    Recommended by:
    Guancous, cardinal, HeyMikey, vcalzone

    I'm thinking about it some more (already commented once in this diary)

    The Democratic primary itself got polled out the wazoo... same methodologies being used today.

    If there were a cellphone factor that was actually giving several points' invisible support to Obama, and away from Clinton, we would have seen it in actual practice.  Instead, the polls ended up being accurate for the most part.

    This means either cellphones were not a factor that threw off polling... or they did, but an offsetting anti-Obama factor (Bradley effect?) threw the polls off in the opposite direction, making them fairly reliable.

    I think the simplest answer (not a factor, generally) is probably the true one here.

  •  I can't get mad at you... (1+ / 0-)
    Recommended by:

    for the Bias thing until we're no longer the defending world champions. :D

    I agree with your final points about likely v. registered voters. Here's something we can all agree on: we HOPE that next year when we study these polls, it turns out that the RV numbers were more accurate. That means the Obama campaign reached their turnout goals. If that happens, we almost can't lose. Big if.

  •  As a professional statistician (6+ / 0-)

    let me say some things:

    First of all, a poll never tells you who is going to win; it gives a snapshot of the electorate on a given day.  If a poll found Obama up by 2% with a 5% MOE, it would still be much more likely that Obama would win than that McCain would win.  To fully explain this gets into Bayesian vs. frequentist statistics, but it's true whichever view you favor.  A poll can't be "completely" wrong, it can only be more or less accurate.

    Second, with regard to cell phones.  All the good pollsters are aware of this problem, and of the demographics you mention.  They account for that by weighting their results per the age breakdown they predict in the actual election, based on previous elections.  So, they've done that.  What they have not accounted for is something more subtle: Are cell-phone users more likely than land-line users of the same age to vote one way or the other?

    Third, you're right about the pollsters not accounting for changing demographics.  I agree with you that more Blacks will turn out this time than in 2004, and most pollsters (all that I've looked at) are not accounting for this.  The key question here is how many more  Blacks will turn out?  And, obviously, this matters more in some states than others.

    I tried to do some of this in race, turnout and Obama's chances but it was pretty crude.

  •  I'm mentally going through a list of people (2+ / 0-)
    Recommended by:
    Guancous, blindyone

    i know that still have landlines.  They are almost exclusively in my parent's age bracket (late fifties/early sixties).  sure enough, they are also split 50/50 between barack and mcinsane, just like the polls.  by that logic, the polls seem an accurate portrayal of only that one demographic, the very same group that barack supposedly has problems with....

  •  Rasmussen contacted me on my land line (3+ / 0-)
    Recommended by:
    Guancous, el dorado gal, kenjisan

    earlier this year.

    This may not be a significant point but both of my young adult children have cell phones (no land lines) and don't answer if they don't recognize the number. Is the whole caller ID thing affecting polling?

    "... he's giving you $5000. in a tax credit- that is the ultimate bridge to nowhere when it costs $12000. to get health care" Joe Biden

    by blindyone on Sat Sep 13, 2008 at 02:16:33 PM PDT

  •  I am not sure what your conclusion is (1+ / 0-)
    Recommended by:

    but this is a very important point, obviously.

    Great diary except, ahem, that Bias bias line, to this old Celtic fan (enjoying his world championships with two teams for the first time in a long life).

    "Ask not what your country can do for you; ask what you can do for your country"

    by Barth on Sat Sep 13, 2008 at 02:20:13 PM PDT

  •  Actually, I believe that Pew found that (2+ / 0-)
    Recommended by:
    Guancous, capelza

    2 to 3 percent of voters were consistently not being sampled because they have cell phones.  Pew then concluded that this is not a significant figure.

    That may not sound like much, but if Obama were to win two thirds of those mostly young voters, that is like the polls consistently understating his results by 1.33 to 2 percent.  I'll take that extra point or two any day.

    It's like those ridiculous poll questions in which two thirds of people say that some major issue will not affect their vote, and the story line is that the issue is not important to voters.  To me, that means that a full third of the electorate may vote on that issue, which is a HUGE deal.

  •  I'm under 30, no landline, got contacted for poll (2+ / 0-)
    Recommended by:
    George, Guancous

    because I happened to be at my parents' house at the time!

  •  I flog this all the time Cells and new voters (1+ / 0-)
    Recommended by:

    The polls are not capturing the enthusiasm of young and new voters
    because they do not reach them.  In 5 years of cell phone use never been called.  No land line for a year but in the past was contacted on that 5 times an election season on my landline  because I live in a liberal mixed wh and aa neighborhood.  I think tweaking for new trends and demographics is in order.  I truly believe that the new voters Obama has pulled in are going to vote and will surprise the chatterers and worriers
    around here.

  •  50 something and no land line (1+ / 0-)
    Recommended by:

    haven't had one in a few years.  

    Back in the days of land lines I always got polled in an election season.  Haven't had a call since I switched over to cell.

    I am guessing that this is a larger problem for pollsters than they want to admit for obvious reasons.  

Subscribe or Donate to support Daily Kos.

Click here for the mobile view of the site