Kos
posted a list of the various poll results from different firms a
couple of days ago to help gauge their relative accuracy. His
said "The verdict? Zogby has been stunningly accurate. SUSA and
ARG have been pretty good."
I posted some extra data in the comments showing that while Zogby was clearly better than SUSA and ARG, he still had some flaws. Most notably in SC and MO, Zogby erred by factors more than acceptable given his MOE.
In the intervening day, I thought about this a little more and
realized the following. Each of Zogby's polls is really 7 little
polls put together (at least with respect to MOE). Becasue of
this, if we look at the data below, Zogby is reporting the results
of 26 different polls. As every poll has the disclaimer that 5%
of the time, the poll will just be wrong, we expect that roughly
one of these numbers is just flat out wrong. Let's take a look at
the data:
NH |
Zogby |
Actual |
Delta |
Delta^2 |
Moe^2 |
Difference |
Clark |
9 |
12 |
3 |
9 |
16 |
|
Dean |
24 |
26 |
2 |
4 |
16 |
|
Edwards |
12 |
12 |
0 |
0 |
16 |
|
Kerry |
37 |
38 |
1 |
1 |
16 |
|
Leiberman |
9 |
9 |
0 |
0 |
16 |
|
|
|
|
|
|
|
|
|
|
|
|
3.74 |
8.94 |
-5.2 |
|
|
|
|
|
|
|
SC |
|
|
|
|
|
|
Clark |
8 |
7 |
-1 |
1 |
16.81 |
|
Dean |
8 |
5 |
-3 |
9 |
16.81 |
|
Edwards |
36 |
45 |
9 |
X |
X |
|
Kerry |
32 |
30 |
-2 |
4 |
16.81 |
|
Leiberman |
5 |
2 |
-3 |
9 |
16.81 |
|
Sharpton |
8 |
10 |
2 |
4 |
16.81 |
|
|
|
|
|
|
|
|
|
|
|
|
5.1 |
8.2 |
-3.1 |
|
|
|
|
|
|
|
OK |
|
|
|
|
|
|
Clark |
31 |
30 |
-1 |
1 |
16.81 |
|
Dean |
6 |
4 |
-2 |
4 |
16.81 |
|
Edwards |
26 |
30 |
4 |
16 |
16.81 |
|
Kerry |
29 |
27 |
-2 |
4 |
16.81 |
|
Leiberman |
6 |
6 |
0 |
0 |
16.81 |
|
|
|
|
|
|
|
|
|
|
|
|
5 |
9.17 |
-4.17 |
|
|
|
|
|
|
|
MO |
|
|
|
|
|
|
Clark |
6 |
4 |
-2 |
4 |
16.81 |
|
Dean |
9 |
9 |
0 |
0 |
16.81 |
|
Edwards |
17 |
25 |
8 |
64 |
16.81 |
|
Kerry |
56 |
51 |
-5 |
25 |
16.81 |
|
Leiberman |
3 |
4 |
1 |
1 |
16.81 |
|
|
|
|
|
|
|
|
|
|
|
|
9.7 |
9.17 |
0.53 |
|
|
|
|
|
|
|
AZ |
|
|
|
|
|
|
Clark |
28 |
27 |
-1 |
1 |
16.81 |
|
Dean |
15 |
14 |
-1 |
1 |
16.81 |
|
Edwards |
7 |
7 |
0 |
0 |
16.81 |
|
Kerry |
42 |
43 |
1 |
1 |
16.81 |
|
Leiberman |
|
7 |
|
|
|
|
|
|
|
|
1.73 |
8.2 |
-6.47 |
|
|
|
|
|
|
|
|
|
|
Overall |
13.66 |
21.28 |
-7.62 |
Now, we see 4 places where Zogby is outside of MOE:
- Edwards in SC, OK, MO
- Kerry in MO
Not a good track record. But, if we remove one of the outliers (say
the 9% error for Edwards in SC), Zogby's predictions get
significantly better. Looking at the SC chart (the "X" shows
where the result is deleted), we see that this brings Zogby's SC
numbers into the overall MOE calculation.
So what's the point? Well, Zogby did better than the others but was
still off more than he should have been, with four results outside
MOE rather than one. Caveat emptor!