(I like these analysis of pollsters. Like anything else, the results change depending on which pollsters you look, and which races you track. I'd love to see this type of analysis for governor races. And for House races as well -- kos)
Ah, American politics. Endless source of meta crap.
Today, I make my contribution to the bloviage as an amateur poll junky. Naturally, I don't just care what the polls say; I want to know what to say about the pollsters themselves.
So I've compiled and processed some data in an attempt to answer the urgent question, "Which pollster, weeks ago, called the best numbers for the 2006 Senate elections, which are, incidentally, over?"
Follow me over the flip as I set forth to settle this pressing matter...
The short answer is, Rasmussen was the most reliable Senate pollster this cycle although, significantly, they didn't perfectly peg a single race. USA Today/Gallup came in a close second. There was a significant drop-off in accuracy to the third-place Survey USA, followed by the much-loved but second-worst pollster, Mason-Dixon. Reuters/Zogby came in last place, despite having exactly pegged more races than any other pollster.
The methodology I employed in reaching this conclusion is the tiniest bit unusual, and, like any such methodology, makes some assumptions.
I didn't merely add up various firms' margins of inaccuracy, because I think that misses the point--no one wants to know whose polls were best in discerning the general landscape; we want to know, on a race-by-race basis, whose polls were the most accurate.
Therefore, I have employed a methodology by which margins of inaccuracy are included, but in which pollsters are also "rewarded" for having the "best poll," "second-best poll," etc. in any given race.
Moreover, I have eliminated from my total sums the "worst" poll by each firm. I figure everyone is allowed an outlier, and in the real world, where real people look at real polls, we can generally spot those immediately and don't give them much weight.
For the record, I don't make any claim to have the Ultimate Methodology. I think, however, that mine is as good as any. My final methodology is this:
1. I looked at five firms (Rasmussen, SUSA, Mason-Dixon, Reuters/Zogby, and USA Today/Gallup) who did extensive polling of Senate races in the final two weeks before the election. I took their final numbers (as reported on RCP) in 11 competitive Senate races.
2. In each race, I figured the margin of inaccuracy for each firm's final poll (taken 10/24 or later) versus the actual numbers.
3. I "ranked" each poll in order from "most accurate" to "least accurate" in each individual race.
4. I subtracted the margin of inaccuracy from a base score of 21 (DISCLAIMER: an arbitrary value) in each race.
5. I further subtracted the "rank" of each poll--for instance, the "first place" poll in each race had 1 subtracted; the "second place" poll had 2 subtracted; etc.
6. I added the final numbers for each major firm.
7. I dropped the lowest score for each major firm (their "outlier").
8. I divided the sum by the number of polls conducted (minus the outlier).
9. I ranked the pollsters from highest average score to lowest.
Incidentally, I compiled the results using some different criteria to see what would make a difference--for instance, I tried including all publicly-available polls (so that the firms wouldn't be compared only to each other, but to the entire polling field), and I tried changing the "21" value up and down a bit. Nothing I did changed the final rankings.
My data follows, with a few thoughts at the bottom.
MISSOURI
ACTUAL: McCaskill +3
FIRM | FINAL POLL | MARGIN | RANK | FINAL SCORE |
Reuters/Zogby | McCaskill +3 | 0 | 1 | 20 |
USA Today | McCaskill +4 | 1 | 2 | 18 |
Mason-Dixon | McCaskill +1 | 2 | 3 | 16 |
Survey USA | McCaskill +6 | 3 | 4 | 14 |
Rasmussen | Talent +1 | 4 | 5 | 12 |
MONTANA
ACTUAL: Tester +1
FIRM | FINAL POLL | MARGIN | RANK | FINAL SCORE |
Reuters/Zogby | Tester +1 | 0 | 1 | 20 |
Rasmussen | Tester +2 | 1 | 2 | 18 |
Mason-Dixon | Tie | 1 | 2 | 18 |
USA Today | Tester +9 | 8 | 4 | 9 |
VIRGINIA
ACTUAL: Webb +1
FIRM | FINAL POLL | MARGIN | RANK | FINAL SCORE |
Reuters/Zogby | Webb +1 | 0 | 1 | 20 |
Mason-Dixon | Webb +1 | 0 | 1 | 20 |
Rasmussen | Tie | 1 | 3 | 17 |
USA Today | Allen +3 | 4 | 4 | 13 |
Survey USA | Webb +8 | 7 | 5 | 9 |
OHIO
ACTUAL: Brown +12
FIRM | FINAL POLL | MARGIN | RANK | FINAL SCORE |
Survey USA | Brown +12 | 0 | 1 | 20 |
Reuters/Zogby | Brown +7 | 5 | 2 | 14 |
Mason-Dixon | Brown +6 | 6 | 3 | 12 |
PENNSYLVANIA
ACTUAL: Casey +18
FIRM | FINAL POLL | MARGIN | RANK | FINAL SCORE |
Mason-Dixon | Casey +13 | 5 | 1 | 15 |
Rasmussen | Casey +13 | 5 | 1 | 15 |
Reuters/Zogby | Casey +8 | 10 | 3 | 8 |
TENNESSEE
ACTUAL: Corker +3
FIRM | FINAL POLL | MARGIN | RANK | FINAL SCORE |
USA Today | Corker +3 | 0 | 1 | 20 |
Rasmussen | Corker +4 | 1 | 2 | 18 |
Survey USA | Corker +5 | 2 | 3 | 16 |
Reuters/Zogby | Corker +10 | 7 | 4 | 10 |
Mason-Dixon | Corker +12 | 9 | 5 | 7 |
MARYLAND
ACTUAL: Cardin +10
FIRM | FINAL POLL | MARGIN | RANK | FINAL SCORE |
Reuters/Zogby | Cardin +5 | 5 | 1 | 15 |
Rasmussen | Cardin +5 | 5 | 1 | 15 |
Survey USA | Cardin +3 | 7 | 3 | 11 |
Mason-Dixon | Cardin +3 | 7 | 3 | 11 |
RHODE ISLAND
ACTUAL: Whitehouse +6
FIRM | FINAL POLL | MARGIN | RANK | FINAL SCORE |
USA Today | Whitehouse +3 | 3 | 1 | 17 |
Mason-Dixon | Chafee +1 | 7 | 2 | 12 |
Reuters/Zogby | Whitehouse +14 | 8 | 3 | 10 |
NEW JERSEY
ACTUAL: Menendez +7
FIRM | FINAL POLL | MARGIN | RANK | FINAL SCORE |
Mason-Dixon | Menendez +7 | 0 | 1 | 20 |
Rasmussen | Menendez +5 | 2 | 2 | 17 |
USA Today | Menendez +10 | 3 | 3 | 15 |
Reuters/Zogby | Menendez +12 | 5 | 4 | 12 |
CONNECTICUT
ACTUAL: Lieberman +10
FIRM | FINAL POLL | MARGIN | RANK | FINAL SCORE |
Survey USA | Lieberman +11 | 1 | 1 | 19 |
Reuters/Zogby | Lieberman +12 | 2 | 2 | 17 |
Rasmussen | Lieberman +8 | 2 | 2 | 17 |
ARIZONA
ACTUAL: Kyl +9
FIRM | FINAL POLL | MARGIN | RANK | FINAL SCORE |
Mason-Dixon | Kyl +8 | 1 | 1 | 19 |
Survey USA | Kyl +13 | 4 | 2 | 15 |
TOTALS
FIRM | RAW SUM | DROP OUTLIER | NUMBER OF POLLS | FINAL AVERAGE |
Rasmussen | 129 | 117 | 7 | 16.71 |
USA Today | 92 | 83 | 5 | 16.60 |
Survey USA | 104 | 95 | 6 | 15.83 |
Mason-Dixon | 150 | 142 | 9 | 15.78 |
Reuters/Zogby | 146 | 138 | 9 | 15.33 |
The most surprising news is how poorly Mason-Dixon stacked up. They really botched a number of races this year. Also interesting about M-D is that their polls this year skewed very much to the right. In fact, in every single race where their accuracy margin was >0, they skewed to the right, sometimes extremely. The lone exception is Arizona (where they underestimated Kyl's advantage by one.)
In other news, Rasumussen called EVERY RACE it surveyed within its 4.5 margin of error. I guess they know what they're doing over there with their relatively small samples and "controversial" methodology...
Alright--sound off, Kossaks!