I got into a heated dispute with a friend recently over the quality of Rasmussen's polling. I even went so far as to dig up Nate Silver's report produced some time back showing that Rasmussen has about a 3% Republican-favorable bias (and that is in their late polling, which almost always drifts towards reality - who knows how horribly twisted their early polls are). Well, I've crunched the numbers. Join me below the fold to see Rasmussen fail again. Please, share this data with any of your right-leaning poll-junkie friends...
I've compiled all of the errors for pollsters polling several races. I looked at only statewide races. And to assure objectivity, I did not want to define which ones to use. So, I used Real Clear Politics website and analyzed every Senate and Governors race that they deemed a toss-up or a "lean". The only exceptions were that I ignored the Rhode Island governor and Alaska senate races, since it is difficult to parse them in terms of determining a Republican or Democratic poll bias (note, I also considered Tancredo as the Republican in the Colorado governor's race).
Here's a table of the data:
Several observations can be made:
- First, a non-Ras observation... KUDOS to all who participated in the GOTV effort. Despite taking a serious hit on Tuesday, EVERY pollster's mean polling bias was Republican-friendly compared to actual results. Considering that undecideds historically go to the challenger, and more Democrats were incumbent than challenger... in THEORY Republicans should have overperformed the polls. In fact, Democrats, bad as the night was, did BETTER than polls suggested (at least at the Senate and Governor levels). I attribute this, at least in part, to a better ground game... getting more of our not-likely voters to the polls. Good job!
- Rasmussen's bias? Wait for it. Wait for it. Wait for it. Out of a total of 30 races, Rasmussen's bias was 3.0% Republican biased! Dead on with their historical record. Note: For future reference, if you want to look at Rasmussen data, it can actually be VERY useful... just always add 3% to the Democrat's column (or subtract 3% from the Republican).
- Out of 30 races, Rasmussen's polling was too Democratic friendly a grand total of four races (MO-Sen, MN-Gov, VT-Gov and TX-Gov). They had zero bias in only three races (KY-Sen, GA-Gov and PA-Gov). They had a Republican bias in the remaining 23 races!! (CT-Sen, CA-Sen, CO-Sen, IL-Sen, NV-Sen, PA-Sen, WA-Sen, WV-Sen, WI-Sen, CA-Gov, NH-Gov, CO-Gov, CT-Gov, FL-Gov, HI-Gov, IL-Gov, MA-Gov, OH-Gov, OR-Gov, AZ-Gov, NM-Gov, SC-Gov and WI-Gov).
- Now, a devil's advocate may say... Well, point #1 may make point #2 less consequential. Ras was, after all, not an outlier, since I just said all polls leaned Republican, right? Wrong, a head-to-head with other pollsters (comparing only mutually-polled races) continues to show Rasmussen with a relatively significant Republican bias:
4a) Versus Quinippiac (6 races). Quin was 0.8% Republican biased. Rass was 1.4% Republican biased in those same races.
4b) Versus SUSA (7 races). SUSA had a 0.1% Republican bias. In those same races Rass was 2.9% Republican biased.
4c) Versus McClatchy/Marist (7 races). Mc/Mar came in at a rather poor 3.3% Republican biased. However, that included the abysmal CO-Gov race, in which Ras did even slightly worse... so, Ras's average for these same seven races was 3.6% Republican biased. That difference is, arguably, within the noise. Nonetheless, strictly by the numbers, Ras still lags McClatchy/Marist.
4d) Versus Mason-Dixon (7 races). MD had some notable errors, but was one of few pollsters with an even split between the number of Dem-biased versus Rep-biased polls. They came in with only a 0.9% Republican bias (driven primarily by their awful polling of the NV-Sen race). Rasmussen tallied a 1.7% Republican bias in these same races. These actually happened to be some of Ras's better polled races, but they too got driven down by the NV-Sen race, and the FL-Gov race hurt them relative to MD (which showed a Democratically biased Sink win).
4e) Versus our own PPP (22 races)!! PPP had some oddly erroneous late-breaking Republican trends, giving them a pretty substantial 2.1% Republican bias. Nonetheless, the "good guys" still beat the "bad guys", as Ras pulled a 3.1% Republican bias in these 22 races.
- Bottom line from point (4) is that Rasmussen ranks worse than EVERY other pollster that polled multiple state-wide races... at least in terms of bias.
- I didn't make a similar spreadsheet with absolute values (magnitudes) just to measure errors. However, here's what the average errors for these six pollsters works out to: Mason-Dixon 4.3%, Rasmussen 4.0%, McClatchy/Marist 3.9%, PPP 3.7%, SUSA 2.7%, Quin 2.2%.
- But again, they don't all have the same sample, so we've got to compare apples-to-apples. So, here's Ras compared to mutually-polled races from the other pollsters:
7a) Versus Quin: Ras error = 1.8%, Quin error = 2.2%
7b) Versus SUSA: Ras error = 3.4%, SUSA error = 2.7%
7c) Versus Mc/Mar: Ras error = 3.5%, McMar error = 3.9%
7d) Versus MD: Ras error = 4.0%, MD error = 4.3%
7e) Versus PPP: Ras error = 3.7%, PPP error = 3.7%
- So, in terms of raw error, Rasmussen seems to be okay... right in the middle of the pack. However, this is not actually a good thing, because it means that it's horrible bias is a systematic (and, likely, intentional) bias. Why can we conclude this? Because if their raw error were worse than all of the other pollsters, then their bias could POSSIBLY be attributed to a few extremely bad polls skewing their results. But this would show up in their error, making them worse than their competition in this regard as well. They're not.
- You can see how systematic their bias is by comparing their tendency to over-favor Republicans. Earlier I note that they were Republican-friendly in 23 of their 30 races polled. That's a 77% rate. How about the other pollsters? Quin leaned too Republican 67% of the time. PPP was too Republican 64% of the time. SUSA 57% of the time. Mc/Mar 86% of the time. MD 43% of the time. Only McLatchy/Marist had a worse rate than Rasmussen, but that's just a matter of sample... Mc/Mar was Republican-friendly in six-of-seven races, and in those same seven races, so was Rasmussen! Moreover, in that 7th race at least Mc/Mar was Dem-friendly... Rasmussen just had zero bias in that race (PA-Gov).
Bottom line, I know I'm preaching to the choir here, but I wanted to back this up with some data... Rasmussen has a systematic, relatively large (for a mean amongst many polls) Republican bias in their polling. It appears to be consistently (on average) around 3% ...Nate Silver observed this in past races and it held true this year. Subjectively, we could argue that earlier in the election cycle it's even larger... but that's tough to gauge. We know, unequivocally, that they have a significant Republican bias in their final polling results.
Other than PPP the other pollsters had a pretty small sample. So, it's tough to say anything definitive, but... SUSA appears to be one of the best out there. Quin's not bad either, esp. after laying some eggs in the last election cycle. McMar has some work to do. Mason-Dixon is good in terms of showing little bias, but they do seem to have a fair amount of noise/error. As for our guys, PPP... their performance was a bit underwhelming but, nonetheless, probably well worth the expense. They polled nearly as many races as Rasmussen, and though their raw error was no better, their bias was a full percentage point better (that's not insignificant when you're talking about 22 polls).
On a final note, to reiterate point #1 ...notice that actual results were more favorable to Dems than ANY pollster showed and contrary to the typical dynamics of undecideds. GOTV efforts work!! Any time we are within a few points in the polls, we can pull it out with the ground game. Remember that when 2012 rolls around and you get bugged to help out with GOTV efforts. Do it! It works!