Skip to main content

There has been considerable discussion in the past day or so about the meaning of the margin of error used by polls to estimate support levels for the various candidate.  I have been saddened to see some real innumeracy in this issue, even among a professed social scientist who said that he works with statistics regularly.

A lot of people seem to be saying that even though Obama had a 5-point lead in the CNN poll, and the margin of error was 3 points, you could not say that this result is statistically significant, since 5 points is not greater than twice the margin of error.

These people are wrong.  The same value can be used for the margin of error for the estimate of each support level as for the hypothesis that one candidate is ahead of the other.  I'll show why after the jump.  The basic thinking is that, when considering the difference between two estimated values, you can ignore one of the tails of the distributions.

Let's make this concrete.  I am a mathematician, so I'm going to mathematize everything.

Let's say there are two candidates C1 and C2, and that a poll claims that C1 has a support level of p1 and candidate C2 has a support level of p2.  And let's say that the polling firm says that the margin of error is espilon at a 95% level of confidence (eq. a p-value of .05).

Most people are with me so far, I hope.  

Let's let  p1 be the estimate of the true level of support for candidate q1, and likewise, p2 is an estimate of the true level of support, q2.  (Ordinarily one would use the same letters and a hat ^ over the variable for the observed value, but I'll do it this way because I'm stuck in ASCII-land.)

The meaning of the margin of error (MOE) is this:
the probability that |p1 - q1| < epsilon is at least 95%.  
(Alternatively, the probability |p1 - q1| > epsilon <.05)</p>

Similarly, the probability that |p2 - q2| <epsilon is at least .95.</p>

What a number of people are doing is using this logic.  If |p1 - p2| > 2* epsilon, and p1 > p2 then the probability of q1 > q2 is also at least .95.

While this is true, this is not a tight bound.  It is true that, if the difference between the two support levels is more than twice the margin of error, that it is easy to see that there is a significant result showing that the leader in the poll is probably the leader in real life.

The logic of the loose bound goes like this: for the person trailing in the poll to be ahead, at least one of the following events must be true:
E1: |p1 - q1| > epsilon
E2: |p2 - q2| > epsilon
Since the probability that either is true is at most .05, the probability that both are true is at most .05, and thus we have a significant result.

But the reason this bound is not tight is because the events E1 and E2 allow for variation that favors the lead!  We are not really concerned about E1 and E2, but rather E1* and E2*

E1*: p1 - q1 > epsilon
E2*: q2 - p2 > epsilon

We can ignore the tails of the distribution on the sides that favor the lead.  Now, since binomial sampling is (approximately) symmetric, if the probability of E1 is .05, then the probability of E1* is half that.  Ditto for E2*.

So Probability(E1*) <.025 and Probability(E2*) < .025.</p>

Now we only see a bad polling result if at least one of E1* or E2* happens.  We can bound the probability that both events have happened by the sum of the probabilities that each event happened separately.

I.e. Probability (q2 > q1) <Probability(E1*) + Probability(E2*) < .05</p>

So if p1 - p2 > epsilon, then it is a significant result.  

What that means for the CNN poll: since the margin of error is 3 points, and the poll shows a 5-point lead, then we can conclude that the lead is statistically significant at 5 points.  It does not have to be 6 points!

Originally posted to RickD on Thu Jul 03, 2008 at 10:16 AM PDT.

EMAIL TO A FRIEND X
Your Email has been sent.
You must add at least one tag to this diary before publishing it.

Add keywords that describe this diary. Separate multiple keywords with commas.
Tagging tips - Search For Tags - Browse For Tags

?

More Tagging tips:

A tag is a way to search for this diary. If someone is searching for "Barack Obama," is this a diary they'd be trying to find?

Use a person's full name, without any title. Senator Obama may become President Obama, and Michelle Obama might run for office.

If your diary covers an election or elected official, use election tags, which are generally the state abbreviation followed by the office. CA-01 is the first district House seat. CA-Sen covers both senate races. NY-GOV covers the New York governor's race.

Tags do not compound: that is, "education reform" is a completely different tag from "education". A tag like "reform" alone is probably not meaningful.

Consider if one or more of these tags fits your diary: Civil Rights, Community, Congress, Culture, Economy, Education, Elections, Energy, Environment, Health Care, International, Labor, Law, Media, Meta, National Security, Science, Transportation, or White House. If your diary is specific to a state, consider adding the state (California, Texas, etc). Keep in mind, though, that there are many wonderful and important diaries that don't fit in any of these tags. Don't worry if yours doesn't.

You can add a private note to this diary when hotlisting it:
Are you sure you want to remove this diary from your hotlist?
Are you sure you want to remove your recommendation? You can only recommend a diary once, so you will not be able to re-recommend it afterwards.
Rescue this diary, and add a note:
Are you sure you want to remove this diary from Rescue?
Choose where to republish this diary. The diary will be added to the queue for that group. Publish it from the queue to make it appear.

You must be a member of a group to use this feature.

Add a quick update to your diary without changing the diary itself:
Are you sure you want to remove this diary?
(The diary will be removed from the site and returned to your drafts for further editing.)
(The diary will be removed.)
Are you sure you want to save these changes to the published diary?

Comment Preferences

  •  "Most people are with me so far, I hope." (14+ / 0-)

    Actually, I was already lost at that point.  But I appreciate that you give us credit for being smart, even if it's undeserved.  :-)

    Now, go spread some peace, love and understanding. Use force if necessary. - Phil N DeBlanc

    by lineatus on Thu Jul 03, 2008 at 10:19:31 AM PDT

  •  I pointed out somewhere else... (0+ / 0-)

    ...that this being true depends on the idea that there are only two options. That isn't the case, because people can respond to a poll by saying "neither" or "undecided". Given that that's a fairly rare option, maybe you're more right than wrong, but maybe not.

    •  hmm (1+ / 0-)
      Recommended by:
      lineatus

      so if Obama leads Clinton by 5 percent and Edwards by 6 percent, and the margin of error is 3 percent...yeah, I could see where my argument wouldn't work.  

      Well, in that case Obama would have a statistically significant lead over each of the others, but maybe not over both at the same time?  It's a problem with multiple hypothesis testing.  

      Don't drink and blog. Think of the children.

      by RickD on Thu Jul 03, 2008 at 10:27:27 AM PDT

      [ Parent ]

      •  To be honest... (0+ / 0-)

        ...I'd call it significant either way (personally, without a scientific basis), just because if the difference is greater than the margin of error even once, then the trailing candidate must be ahead and the leading candidate behind, neither of which can logically be more than a 50% shot (and much less than 50% to produce a variation near the tail end of the curve of possible errors), and so both together must logically be a chance of 25% or (more likely) less. So, y'know, I tend to agree that if one candidate leads by more than the margin of error, it's very probable that they're actually leading.

  •  hope people like the math (8+ / 0-)

    If not, just take home the message that it's OK to use the margin of error in the way that seems like it would be correct.  If the difference between two candidates is greater than the margin of error, then it's a statistically significant lead.

    Don't drink and blog. Think of the children.

    by RickD on Thu Jul 03, 2008 at 10:24:35 AM PDT

    •  I consider myself a reasonably intelligent man (0+ / 0-)

      but you lost me when candidate C1 maybe became candidate q1.  The "p"s and "q"s and epsilons were interesting, but your last paragraph nicely summed it up for me.  I read the statistician's take on margin of error and it didn't make any more sense than this.  But I always felt that margin of error means "margin" of error, so if the difference was greater than the "margin" of error, then that candidate must be leading, it just makes common sense.  I'm very much into common sense, but MSM and rethugs seem to be diametrically opposed to common sense and in fact look for the things that make the least sense for their position.  I'm glad you cleared this up.  I'd love to see a debate between you math dudes and you statistics dudes (or dudesses, don't want to be sexist here) to battle it out in mathematically combat.  No violence now.

      "War is Peace, Freedom is Slavery, Ignorance is Strength", George Orwell, "1984" -7.63 -5.95

      by dangoch on Thu Jul 03, 2008 at 10:51:15 AM PDT

      [ Parent ]

    •  doubling the MOE (1+ / 0-)
      Recommended by:
      lineatus

      changes the confidance level from 95% to something more like 99.9%. If the lead is greater than double the MOE, not only can you conclude the lead is significant (which you can do without doubling), you can conclude the lead is a slam dunk - if the election were held the day of the poll.

  •  I'm not a mathematician but your last paragraph (3+ / 0-)
    Recommended by:
    theran, Shirl In Idaho, Greasy Grant

    clarifies the matter for me.

    "What that means for the CNN poll: since the margin of error is 3 points, and the poll shows a 5-point lead, then we can conclude that the lead is statistically significant at 5 points.  It does not have to be 6 points!"

    Thanks. To put it differently, CNN marginalized or diminished the significance of Obama's lead while increasing the significance of McCain's second place positioning in the polls.

    We must use time creatively... and forever realize that time is always ripe to do right. Martin Luther King, Jr.

    by Jezreel on Thu Jul 03, 2008 at 10:25:57 AM PDT

    •  CNN and the rest of them (1+ / 0-)
      Recommended by:
      Jezreel

      are going to report this as a "virtual tie" all the way through the election.  If they can't paint this as a horse race then what the heck will they talk about?  I mean, people seem to be tired of Britney, Lindsay, and Paris trash talk and a million jerks chasing them with cameras.

      *the blogger formerly known as shirlstars

      by Shirl In Idaho on Thu Jul 03, 2008 at 11:37:06 AM PDT

      [ Parent ]

      •  Shirl you may have accurately stated CNN's (1+ / 0-)
        Recommended by:
        Shirl In Idaho

        motives. I would also add, however, that CNN would prefer that John McCain wins the election. I say this because when Clinton was ahead in the polls by 5 or fewer points, her lead was never reported as a virtual tie with Obama. Instead, it was portrayed as a possible way for her to win the nomination.

        We must use time creatively... and forever realize that time is always ripe to do right. Martin Luther King, Jr.

        by Jezreel on Thu Jul 03, 2008 at 11:52:24 AM PDT

        [ Parent ]

  •  Great work (0+ / 0-)

    I've wanted to spill the beans about statistical methods, but I can't put math into words like you just did here.

  •  Linked numbers (0+ / 0-)

    I couldn't follow your numbers beyond the most general sense, but it occurs to be that there is a flaw in that these numbers are tightly linked.  If there is a 5% chance that McCain's actual numbers are 3 points higher, then it doesn't really matter what the percentage chance of Obama's number being right is.  If McCain is 3 points higher, Obama is 3 points lower more or less by definition.

    Thus, because the two probablities are linked, your analysis fails and if the numbers are within twice the margin of error, you can't claim to be in the clear.

    •  doesn't matter (0+ / 0-)

      If Probability(E1*) < .025 and Probability(E2*) <.025, then Probability (E1* or E2*) < .025 + .025.  That inequality holds regardless of any correlation between the two events.</p>

      In fact, any correlation between the two events only makes the bound stronger.  In the worst case, the two events are anti-correlated.  

      Since, as you point out, it is far more likely that the two events are positively correlated, then we probably could get away with shrinking the margin of error.

      Don't drink and blog. Think of the children.

      by RickD on Thu Jul 03, 2008 at 10:34:21 AM PDT

      [ Parent ]

      •  Your logic is backwards. (0+ / 0-)

        I don't think that holds.  Since the chance that McCain could be 3 points higher is WITHIN the margin of error (and thus of significantly greater chance than the .025 you posit), it doesn't matter what the chance of Obama's numbers being off are.  If McCain's go up, Obama's perforce come down.  You don't need to calculate it at all (it's like saying: the chance of me rolling a 6 on a die are equal to: the chance of rolling a 6 plus the chance of not rolling 1-5.  But these are the same thing).  

        •  The example was 50-45 (0+ / 0-)

          You're not analyzing the d.o.f. exactly here either.

          I'd split the difference: you are ignoring that there is some amplification from symmetry and Rick needs a slightly tighter argument.  

          He'll lose a little, but not a giant amount.

          Ortiz/Ramírez '08

          by theran on Thu Jul 03, 2008 at 10:47:41 AM PDT

          [ Parent ]

      •  but (0+ / 0-)

        P(E1*) </ 0.25, P(E1*)= 0.25, per your logic above.</p>

        At which point, you have disregarded possible skewing toward the lead end, which could be < or > eps/2. You can't necessarily assume straight normal dist. in all circumstances, especially if there is weighting involved. Gaussian is assumed, but not necessarily accurate.

  •  I agree it's not double, but this seems wrong too (2+ / 0-)
    Recommended by:
    theran, Feanor

    I agree it's not double (or that double isn't a tight bound, anyway), but except to a pretty rough approximation it's not just the margin of error either.

    The standard way I've been taught to do this is to calculate a standard error for the difference |p-q|, and test the hypothesis that this is positive or negative. This is larger than the standard error for either p or q, though not twice as large.

    The standard approximation I usually use is err(|p-q|) = sqrt(err(p) + err(q)). So, for example, if the standard error on each of p and q is 5%, then the standard error for the difference is around 7%.

    In this case there's an additional correction since p and q are probably pretty close to perfectly negatively correlated. With that assumption, Wikipedia gives a formula for computing the standard error of the difference directly from the observed percentages and sample size, as: sqrt((p(1-p) + q(1-q) + 2pq) / n).

    Disclaimer: I'm not a statistician, although I am a computer scientist who works (sometimes) in the area of statistical machine learning.

    "See a world of tanks, ruled by a world of banks." —Sol Invictus

    by Delirium on Thu Jul 03, 2008 at 10:34:38 AM PDT

    •  your logic makes much more sense. (0+ / 0-)

      per my post below, at the very least, the difference between two independently reported values is statistically significant if it is, at minimum, greater than the reported MOE for each value, that is, >3.5, but never >=3.5. Even then, typically the difference between the two must be ~2*MOE to be statistically sig, but, as you  point out, it can be lower, depending on reported values of "other", etc.

    •  hmm and actually it might be double (0+ / 0-)

      If there are exactly two candidates and no other options so the two figures are perfectly negatively correlated, then I think it is double the MOE. Given a value for a, and the information that b = 1.0-a, then the question "what is the margin of error for |a-b|?" is identical to "what is the margin of error for |a-(1-a)| = 2a-1?". And the margin of error for 2a is twice the margin of error for a.

      "See a world of tanks, ruled by a world of banks." —Sol Invictus

      by Delirium on Thu Jul 03, 2008 at 10:53:18 AM PDT

      [ Parent ]

    •  cannot find the wiki formula (0+ / 0-)

      though I've seen it used elsewhere.  I've been trying to reconcile this line of argument with what I've been saying, and I'm not making progress.

      Ok, for starters, we're talking about (p-q), not |p-q|, right?  The latter is never going to be negative.  :)

      We are interested in: what is the area under the curve of the distribution of (p-q) below the value of zero?

      Ah - I see the connection now.  We only have to look at one side of the normal distribution.  That could account for the disappearance of a factor of two.

      I still don't see a flaw in my argument, though.

      Don't drink and blog. Think of the children.

      by RickD on Thu Jul 03, 2008 at 02:12:28 PM PDT

      [ Parent ]

  •  Tips for use of the word "innumeracy" (0+ / 0-)

    One of my most posted topics. :-)

    Eli Stephens
    Left I on the News

    by elishastephens on Thu Jul 03, 2008 at 10:35:17 AM PDT

  •  What I want (1+ / 0-)
    Recommended by:
    theran

    I want polls to report the probability that the leading candidate is really ahead.  I think that it's calcuable, but I don't know how to do it.  But if we have a poll that says Obama 50 - McCain 45, 3% Margin of Error, I want to know what the percentage chance is that O>M.  I don't care if it's 10 points over or .0001% over.  Just that he's on top.

    Right on, Dr. Dean.

    by Mikey on Thu Jul 03, 2008 at 10:35:31 AM PDT

    •  yeah (1+ / 0-)
      Recommended by:
      theran

      The p-value of that poll should be easy to calculate.
      I would also prefer it if polls were presented in that manner, but I think there is a sentiment that prefers using the margin of error as a clunky replacement for 'significant result' as opposed to 'insignificant result'.

      Don't drink and blog. Think of the children.

      by RickD on Thu Jul 03, 2008 at 10:39:06 AM PDT

      [ Parent ]

    •  This is the correct language (0+ / 0-)

      But apparently the software pollsters use doesn't support this.

      Ortiz/Ramírez '08

      by theran on Thu Jul 03, 2008 at 10:39:08 AM PDT

      [ Parent ]

  •  A Few Things (1+ / 0-)
    Recommended by:
    weasel
    1. The MOE was 3.5 not 3.
    1. The dead heat reference could be in regard to the poll when Nader and Barr are included, then the lead is only 3 points not 5.
    1. Without trying to decipher your post, the fact remains: the MOE refers to each candidate's percentage and not the difference between them. Therefore, any lead less than double the margin of error could mean that the candidate behind is actually tied or slighly ahead, although the probability is greatly reduced outside the "inner" margin of error.
    •  read the diary again (1+ / 0-)
      Recommended by:
      Mikey
      1. is not terribly relevant.

      As for 2. I would presume that both Barr and Nader have vanishingly small levels of estimated support.  One nice thing about that is that the margin of error decreases when the parameter being estimated itself decreases.  In practical terms, that means that if the MOE for an esimate of 40% is 3 points, then the margin of error for estimate 4% is less than point.  

      As for 3, well...if you want to substitute a layman's understanding of what MOE for an expert's understanding, go ahead and do so.  Your comment essentially consists of 'I didn't understand your diary, but I'm going to insist you are wrong anyway'.

      shrug

      Don't drink and blog. Think of the children.

      by RickD on Thu Jul 03, 2008 at 10:51:47 AM PDT

      [ Parent ]

  •  actually (0+ / 0-)

    you either have a typo or flawed logic, because you state:

    We can ignore the tails of the distribution on the sides that favor the lead.  Now, since binomial sampling is (approximately) symmetric, if the probability of E1 is .05, then the probability of E1* is half that.  Ditto for E2*.
    So Probability(E1*) <.025 and Probability(E2*) < .025.</p>

    but in fact, you mean, P(E1*) = 0.025 and P(E2*) = 0.025, not less than.

    Also, there is a hole in your argument when you claim that P(q2-q1=0) <P(E1*)+P(E2*), because you can't legitimately add between probabilities, to obtain the LHS. At very best, you can say that the lead is significant>3, but in no way is =3 allowable, by definition of Max Margin of Error.

    (I was a stats major in undergrad)

    •  why less than (0+ / 0-)

      I thought I had been using "<" instead of "=" consistently.</p>

      In most statistical studies, what is reported as "the margin of error" is in fact an approximate value.  For starters, it is typical to approximate the binomial distribution with its normal approximation.  And then, when looking at the variance of the normal approximation, there is probably more approximation going on.

      Furthermore, I am telling a small lie when I say that the distribution is symmetrical.  It is only very close to symmetrical.  

      And that's why I use "<" instead of "=".  I really didn't mean "=".</p>

      As for your second complaint, you certainly can add the probabilities.  If you have two different events E1* and E2*, the probability that at least one of the two events has happened is at most the sum of the independent probabilities of each event.  This is practically axiomatic.  And it is immediate from set theory.

      Pr(E1* or E2*)= Pr(E1*) + Pr(E2*) - Pr(E1* and E2*)

      And thus if Pr(E1* and E2*) > 0,
      Pr(E1* or E2*) < Pr(E1*) + Pr(E2*).

      Don't drink and blog. Think of the children.

      by RickD on Thu Jul 03, 2008 at 10:59:13 AM PDT

      [ Parent ]

  •  Don't forget the "likely voter", & other problems (1+ / 0-)
    Recommended by:
    Feanor

    The problem with applying pure mathematical analysis to the political polling problem is the need/requirement for "modeling." In the classic probability example, we pull white or red balls out of a large jar, and just need to ascertain how the sample we pull out represents the complete contents of the jar. But in political polling, some of those balls may not "count" because they aren't actually going to vote, so a model has to determine, based on other questions, if they are or aren't.

    Then there's the sampling problem. In the jar problem, we assume the balls are evenly distributed. But what if all the black balls are on the top and we don't reach down too deeply. Clearly the "answer" we arrive at is incorrect. Similarly, the political poll has to somehow sample people "evenly" when those people are almost always distributed unevenly (e.g., inner cities vs. suburbs, etc.) with differing problems (e.g., young people with cell phones vs. working people away from home vs. older people at home answering their phones).

    It's not just a mathematical problem, alas.

    Eli Stephens
    Left I on the News

    by elishastephens on Thu Jul 03, 2008 at 10:47:34 AM PDT

  •  What ever the poll says add 10 to McSame (0+ / 0-)

    let's not encourage the tortoise hare syndrome, whatever the poll says work harder, run faster, rest in  December!

    If you could have any superpower what would it be? The Big Funny Live

    Obama and the Racists "Against the assault of laughter nothing can stand." - Mark Twain

    by THE BIG FUNNY on Thu Jul 03, 2008 at 10:55:14 AM PDT

  •  Statisticians generally use a 95% confidence... (0+ / 0-)

    interval as a threshold of 'statistical significance'.  What CNN and the rest of us need is a definition of a 'statistical dead heat'.  If survey results are so close that the leading candidate has a 49% chance of actually trailing among the full population, we can safely say that the race is tied at that moment.  I'd be comfortable saying that if the leading candidate has a less than 35% chance of actually trailing, the lead is significant enough so as not to be a statistical dead heat.  Any other thoughts out there?

    Yes, I realize I should get a life.  Whenever I hear the song Chances Are and Johnny sings the line "the chances are your chances are awfully good, I think to myself, "What the f*ck does he mean by that?"

    "Some men see things as they are and say 'Why?' I dream things that never were and say, 'I need to quit drinking!'" - Greasy Grant

    by Greasy Grant on Thu Jul 03, 2008 at 11:05:23 AM PDT

  •  Brilliant line (0+ / 0-)

    I am a mathematician, so I'm going to mathematize everything

    Well done analysis as well thanks for taking the time.

  •  For once, a scientific explanation. (0+ / 0-)

    Takes me right back to my undergraduate statistics class.  This man speaks the truth!

    Don't lend your hand to raise no flag atop no ship of fools.

    by mojave mike on Thu Jul 03, 2008 at 11:52:35 AM PDT

Subscribe or Donate to support Daily Kos.

Click here for the mobile view of the site