Skip to main content

Small thing that's really starting to annoy me....

Like most of you, I was excited to see that groups are rating the performance of polls.

But these rating systems are using the current national popular vote count to measure pollster performance. Or...the national popular vote from 10am this morning. Or God knows when.

The problem is that using this morning's national popular vote margin is wrong. The national popular vote will shift toward Obama quite a bit. Nate Cohn (TNR's Electionate) estimated this morning that it could shift as much as 1.2%.

It's obvious why this is the case: California. But it's not just California. There are a ton of ballots to count - many of which are actually in states that the networks are reporting at 100%.

At any rate - if you're measuring pollster performance using an outdated national popular vote margin as your baseline, you're probably not fit to measure pollster performance.

I don't write this to take away any of the excitement from PPP's performance. In fact, they may still end up as one of the best pollsters. Their final poll had it at Obama +3, and he'll probably end up at +3.4 at the most.

Think of it this way: if this data were run with accurate numebrs, Rasmussen would be EVEN WORSE.

Your Email has been sent.
You must add at least one tag to this diary before publishing it.

Add keywords that describe this diary. Separate multiple keywords with commas.
Tagging tips - Search For Tags - Browse For Tags


More Tagging tips:

A tag is a way to search for this diary. If someone is searching for "Barack Obama," is this a diary they'd be trying to find?

Use a person's full name, without any title. Senator Obama may become President Obama, and Michelle Obama might run for office.

If your diary covers an election or elected official, use election tags, which are generally the state abbreviation followed by the office. CA-01 is the first district House seat. CA-Sen covers both senate races. NY-GOV covers the New York governor's race.

Tags do not compound: that is, "education reform" is a completely different tag from "education". A tag like "reform" alone is probably not meaningful.

Consider if one or more of these tags fits your diary: Civil Rights, Community, Congress, Culture, Economy, Education, Elections, Energy, Environment, Health Care, International, Labor, Law, Media, Meta, National Security, Science, Transportation, or White House. If your diary is specific to a state, consider adding the state (California, Texas, etc). Keep in mind, though, that there are many wonderful and important diaries that don't fit in any of these tags. Don't worry if yours doesn't.

You can add a private note to this diary when hotlisting it:
Are you sure you want to remove this diary from your hotlist?
Are you sure you want to remove your recommendation? You can only recommend a diary once, so you will not be able to re-recommend it afterwards.
Rescue this diary, and add a note:
Are you sure you want to remove this diary from Rescue?
Choose where to republish this diary. The diary will be added to the queue for that group. Publish it from the queue to make it appear.

You must be a member of a group to use this feature.

Add a quick update to your diary without changing the diary itself:
Are you sure you want to remove this diary?
(The diary will be removed from the site and returned to your drafts for further editing.)
(The diary will be removed.)
Are you sure you want to save these changes to the published diary?

Comment Preferences

  •  Ayup (2+ / 0-)
    Recommended by:
    Iberian, sebastianguy99

    Nate normally does his own self-evaluation once the polls are DONE.  At the same time he discusses the other aggregators and pollsters.

    Think I'll wait till then.

    By my quick look, it seems to me that 538 did better on the president, TPM on the senate races.

    Though I am far more interested in seeing how close to the final vote percentages were for various aggregators rather than specific state win/lose predictions. (I would consider an aggregator more accurate if they predicted a .2% Obama loss in Florida than an aggregator who predicted a 3 point win for Obama in Florida).

    "Empty vessels make the loudest sound, they have the least wit and are the greatest blabbers" Plato

    by Empty Vessel on Wed Nov 07, 2012 at 03:03:23 PM PST

  •  Yes (2+ / 0-)
    Recommended by:
    Gooserock, yella dawg

    This morning you had people like Bob Schieffer saying that it was a squeaker in the pop. vote. No it is not.
     He was not the only one. When even the politicos don't freaking know or understand the voting system in the USA it is clear that we have some dumb system, and yes also dumb politicos, but that's generally a given

    •  yeah but the same dumb people (1+ / 0-)
      Recommended by:

      were saying the election was "dead even" when the polls showed electorally speaking, Romney had a VERY difficult path to 270.  So why should it surprise anyone they continue to be wrong. They just need something to say otherwise it would be dead air.

  •  I think the need to be evaluated at 3 different (1+ / 0-)
    Recommended by:
    Theodore J Pickle

    Times as well.  Before the debates, after them all, and Election Day, to show who's got a clue about predicting.
    Else why have them early?

  •  I think we'll see the aggregators featured more (0+ / 0-) upcoming cycles. This idea of using any ol' poll to drive the day's narrative was shown to be a foolhardy approach.

    "There is nothing more dreadful than the habit of doubt. Doubt separates people. It is a poison that disintegrates friendships and breaks up pleasant relations. It is a thorn that irritates and hurts; it is a sword that kills.".. Buddha

    by sebastianguy99 on Wed Nov 07, 2012 at 03:20:19 PM PST

  •  I ran some quick numbers (0+ / 0-)

    I ran some quick numbers on the pollsters in the 12 battleground states (the firewall - OH, WI, IA, NV, the Hail Mary states - PA, MI, MN, and the other battlegrounds - FL, NC, VA, CO, NH) that did at least 3 polls in these states.  If the pollster also did a national horse race, I included that, too.  I used the margins as of about 9 AM this morning.  I did not include Gravis since they are actually a fraud,not a pollster.

    I measured performance on three metrics - accuracy (how many of the states were called in the right direction), bias (was the average error in one direction or the other), and standard deviation.  For accuracy, if the pollster had the race as a tie, I counted that as "corrrect" if the vote margin was less than 1% (i.e. FL...).

    PPP scored really well on accuracy hitting 11 of 12 states plus the national.  Rasmussen, OTOH, hit 5 of 11 states (they didn't poll MN) and missed the national.  PPP's bias was 0.7% Republican, and their SD was about 2.5%, which were both middle of the road.

    On accuracy, NBC/Marist hit on 6 of 6 states plus the national.  PPP and YouGov hit 11 of 12 states plus the national.  Mellman hit on 5 of 5 states, and CBS/Quinnipiac hit on 4 of 4 plus the national.  The worst by far was Rasmussen with 7 misses in 12 polls.

    On bias, the two overtly Dem firms, Mellman and Grove Insights, were the only 2 that had zero bias in their final results.  Quinnipiac was the only credible pollster to have a Dem bias (0.3% - Zogby did as well, but he is not credible IMO).  Rasmussen's bias was 4.3% GOP, Suffolk 4.0%, and ARG 4.8%.

    On SD, Quinnipiac was the lowest at 1.7%, followed by Angus-Reid 2.0%, and CNN/ORC & Ipsos 2.1%.  The usual suspects were at the bottom of the list - Rasmussen 4.8%, ARG 4.9%, and the worst of the worst Suffolk with a standard deviation of 5.3% from the actual margin.

    So, there you go.  NBC/Marist hit the mark most efficiently, the Dem firms Mellman and Grove were the least biased, and Quinnipiac had the lowest SD.

    I ran the numbers on Nate as well, just to compare.  Obviously, he would get the accuracy prize for hitting 13 of 13 since he got everything right.  His bias was 0.6% Republican, which is only middle of the pack, and his SD was 1.6%, which would top Quinnipiac for the lowest by 0.1%.

  •  all that 100 percent means at this point is (0+ / 0-)

    that 100 percent of precincts have reported some numbers.

    Depending on how many vote by mail voters there are in any state, or county, there can be huge numbers of uncounted ballots.

    In my county, they were so busy they didn't even count ballots that arrived on Monday. then there's the uncounted ballots that arrived in Tuesday's mail. THEN, there's all the vote by mail ballots dropped off at the polls -- we have many in our county because many precincts were switched to all mail-in voting, and many of those voters don't want to vote early, so they drop their ballots off.

    We've had as high as 45 percent of ballots not counted in final Election Night report -- even tho the online results show "100 %" reporting -- and this year, I expect (from what the voter registrar has told us) that it will be at least a bit higher than that. We won't get final results here until the election is certified, which is usually at least 3 weeks after Election Day. Some voter registrars report final results before the election is certified. But in any case, it's way way way too early to be talking numbers or percentage of turnout.

    Media should remember this from prior years, but they dont seem to. Back in the day, there were only a few absentee ballots or provisional ballots to count after election day. It's different now. But, only the very closest of elections are likely to be changed by late reported ballots: Prior elections show us that the late counted ballots come in at more or less the same percentages as earlier counted ballots.

  •  Polling on Polls = Meta Madness?? (0+ / 0-)

    I agree on your main point -- if they're still counting, some margins are suspect, but if you'll let me go all meta here....

    Having obsessed about polls for months, do we get to go meta here and start doing polling about polling?  How madly meta is that?

    [I]t is totally not true that Mitt Romney strapped Paul Ryan to the top of a car and drove him to Canada. Stop spreading rumors! -- Gail Collins

    by mbayrob on Wed Nov 07, 2012 at 04:37:36 PM PST

  •  I think Obama lost a couple of million votes (1+ / 0-)
    Recommended by:

    from east coast turnout being way down due to Sandy.  New York and New Jersey were way off their 2008 totals.

  •  Cali, but even more OR and WA 100% vote by mail (0+ / 0-)

    -1.63/ -1.49 "Speaking truth to power" (with snark of course)!

    by dopper0189 on Wed Nov 07, 2012 at 05:41:43 PM PST

Subscribe or Donate to support Daily Kos.

Click here for the mobile view of the site