How large is opinion poll "bias" in presidential approval surveys?
It's not necessary to spend a lot of time studying surveys to notice that often polls on presidential approval taken at roughly at the same time report different results. Here on DKos a folklore of sorts has grown up around these surveys, with some polls (e.g., Rasmussen) viewed as regularly "leaning" towards Bush (reporting higher approval numbers), while others (e.g. Zogby) seen as "leaning" away from him.
Recently political scientists Nathaniel Beck, Simon Jackman and Howard Rosenthal (BJR) reported the results of an unusual study of presidential approval -- "Presidential approval: The case of George W. Bush" -- that sheds a great deal of light on disparate opinion survey results. (Google the three authors to find their paper).
BJR analyzed the reported Bush approval scores of 1,111 polls on pollingreport.com; they supplemented these with 28 others. Both their poll data and other variables (gasoline prices, unemployment claims, casualties in Iraq) were measured at weekly intervals. This permits a much more fine grained analysis than using months or quarters as the time period. (They also have a method to deal with pivotal events like 9/11, but that discussion assumes a lot of technical knowledge). As a byproduct of their effort to estimate a statistical model of popularity, they generated estimates of the "bias" of each pollster in their study.
Before viewing their results, remember that "bias" is defined statistically: it's the deviation of a poll from the trendline fit to the data by statistical analysis. It does NOT imply anything about pollsters' intent or their ideology -- it's just the end result of a series of practical decisions made by pollsters about how to conduct a poll. Different polling methods yield different polling results. It also is not a comment on the reliability or usefulness of the results. (If you have a clock that you know is always 4 minutes fast, it's just as good a time-keeper as one that always has the correct time).
Below are their results. For each pollster, I present the number of their polls in the study and their average deviation from the trendline. Positive numbers indicate a lean towards Bush, negative ones away. Thus, Rasmussen is represented by 14 polls (all during or after the spring of '05, BTW), and on average their approval scores for Bush are 3.7% higher than the trendline. Where numbers have been omitted by BJR, I approximate a value from their graph. (For pollsters that contribute a small number of polls a statistical estimate of "bias" is not meaningful, but you can find them on the graph in their paper if you're curious).
Rasmussen n=14, +3.7%
Annenberg n = 23, +3.3
Diageo Hotline n=12, +2.8
ABC-WaPo n=83, +2.5
Cook n=7
NPR n=17, +1.9
Time n=22, approx. +1.7
CNN-USAToday-Gallup n=202, +1.6
LA Times n=21, +0.9
Fox n=113, +0.7
WNBC n=4
USAToday-Gallup n=4
CNN n=4
CBS n=105, -0.4
NBC/WSJ n=49, -0.6
Harris n=46, approx. -0.8
AP/IPSOS n=79, approx. -1.2
Newsweek n=77, -1.5
Time-CNN n=32, approx. -1.7
Pew n=74, approx. -1.8
Quinnipiac n=25, approx. -2.1
Zogby n=78, -2.8
Bloomberg n=11, -3.0
CSN n=31, -6.2
There's more to report about this study, but first I'll wait to see if there is any interest in this.