With the new ARG poll out there are questions about the relative importance of sampling error and model error. Several commentators have mentioned that it is likely to be model error that is causing the wide disparity in results rather than sampling error. This is almost certainly true, but many dont know what the difference between the two. Here it is:
SAMPLING ERROR - usually reported as the "margin of error" tells you what confidence you have that the value computed from the sample you picked actually represents the true value in the population as a whole, ASSUMING THAT YOU HAVE THE CORRECT MODEL FOR HOW THE CHOICE IS MADE IN THE FIRST PLACE. For example, if we look at coin tosses, the "correct model" is that there is a 50-50 chance of heads vs. tails. If I toss the coin ten times I may or may not get this 50-50 split from my sample of tosses. The more tosses, the more likely I am to get something close to 50-50 again ASSUMING THAT I AM RIGHT THAT THE CORRECT MODEL IS 50-50.
The usually reported "moe's" are correctly understood to mean the following: If I repeated my sample 20 times on the same population, then 19 times out of 20 the "true" value for the number lies within the range given by my estimate plus or minus the moe.
MODEL ERROR - is far more important when we dont really know the true nature of the model and are only guessing at it. The "true" parameter for Bush vs. Kerry is whatever percentage we would get if we could actually ask every single person who really is going to vote in November who they would vote for and they actually told us the truth and then actually did go vote that way. Where could this go wrong in an actual poll?
Read More