Skip to main content

View Diary: OK: Legit civics test shows students know George Washington (108 comments)

Comment Preferences

  •  You haven't been reading Nate Silver (13+ / 0-)

    He has explicitly raised the possibility that Strategic Vision is simply fabricating all their numbers. He doesn't make that accusation lightly, and having seen the evidence, I'm extremely suspicious of SV.

    He looked particularly at the results from the OK school survey. The case that they were faked is actually quite strong: if you look at the distribution of the overall student scores, it matches very closely the distribution you get if you assume that all the questions are entirely independent. That's highly unlikely to be true: some students are smarter than others, or like social studies better, or work harder, and that increases their chance on every question - the student scores should match a model which assumes a degree of correlation.

    In America, 60% of bankruptcies are because of medical bills, and 80% of those people had health insurance

    by sullivanst on Thu Nov 05, 2009 at 10:33:18 AM PST

    [ Parent ]

    •  In his commentary (3+ / 0-)

      on the original OCPA survey, Nate Silver never said that Strategic Vision faked any answers or faked the survey. The harshest thing he said was that there is something "very funny" about Strategic Vision's survey process in general.

      Based on what one knows about how the original Oklahoma survey was conducted, it doesn't require jumping to any conclusions about the survey being faked. Based on what we know, the survey was conducted by telephone, possibly automated. Imagine you are sitting at home. You are a ninth grader. You're deeply absorbed in some TV show. You get a call. It's an automated survey. You decide to do the survey. You are going through the questions.

      One question goes like this:

      "Who was the first president of the United States? Press 1 for George Washington, press 2 for Abraham Lincoln, press 3 for Thomas Jefferson, press 4 for Franklin Roosevelt, press 5 for John Adams, press 6 for George W. Bush, press 7 for Barack Obama, press 8 for Richard Nixon, press 9 for John F. Kennedy, press 0 for Don't know."

      It doesn't take a genius to figure out how the results might be flawed from such a survey. Half of them probably forgot which number to press, the other half probably just pressed some random number to get to the next question. It doesn't require faking anything. Just bad methodology.

      •  Uh, no. (3+ / 0-)
        Recommended by:
        HudsonValleyMark, the karina, Tm3

        You didn't read Silver very carefully.

        He hedged slightly, for obvious legal reasons, but he pretty much said they made the whole thing up. The first hint's in the title of the post: Are Oklahoma Students Really This Dumb? Or Is Strategic Vision Really This Stupid?. (emphasis mine)

        Then later on he says:

        Let's run a couple of simulations to test the robustness of these results. In the first simulation, I'll assume that:

        (i) the student body is homogeneous -- everyone is as knowledgeable as everyone else, and
        (ii) the questions are independent of one another; so knowing, say, who wrote the Declaration of Independence doesn't make you any more (or less) likely to know what the Bill of Rights is.

        These are completely unrealistic assumptions, which, as you'll see, is the whole point.

        Nate's emphasis, that time.

        In conclusion he says (emphasis mine again):

        It seems quite strongly possible, nevertheless, that the students polled for this survey don't exist anywhere in Oklahoma but instead on a hard drive somewhere in Atlanta.

        Here he's directly stating there's a strong possibility Strategic Vision entirely fabricated the results.

        As for the survey's claimed methodology, OCPA's press release strongly implies it was live callers:

        In Oklahoma, the telephone surveyors called a sample of 1,000 public high-school students and read the following statement: "On the next 10 questions, I will be asking you questions about American government and history. Give me your best answer, and it is permissible to respond ‘I don't know.'"

        This was not a multiple choice test, it was a free answer test. And that's how the paper version was also given - the only significant difference between the phone version and the paper version, if you assume that both were actually performed, was whether the environment was conducive to focussed students, and grade level. I don't see grade level as being that significant - OK students are supposed to have been taught all the answers to those questions before reaching high school, as OCPA's press release pointed out.

        In America, 60% of bankruptcies are because of medical bills, and 80% of those people had health insurance

        by sullivanst on Thu Nov 05, 2009 at 11:52:47 AM PST

        [ Parent ]

        •  oops... forgot to emphasize... (0+ / 0-)

          in the middle blockquote, I meant to emphasize:

          It seems quite strongly possible, nevertheless, that the students polled for this survey don't exist anywhere in Oklahoma but instead on a hard drive somewhere in Atlanta.

          In America, 60% of bankruptcies are because of medical bills, and 80% of those people had health insurance

          by sullivanst on Thu Nov 05, 2009 at 12:04:45 PM PST

          [ Parent ]

        •  There are some (1+ / 0-)
          Recommended by:
          BYw

          serious questions about the original survey, no doubt about it. It is not clear whether it was actually multiple choice or free answer. The way it is described sounds kind of ambiguous. And with the results they reported, some of the answers don't seem consistent with a free answer test.

          For example, the question on "What are the two political parties in the U.S.". How could 11% answer "Communist and Republican", but EVERYONE else answered either "Democrat and Republican" or "Don't know"? Out of 1,000 survey respondents, 111 said "Communist and Republican", but no one else said anything other than "Democrat and Republican" or "Don't know"? I don't see how that result is even possible with a free answer test. The only way I can see getting that result is through multiple choice.

          And if the original survey was done using live callers rather than automated, that makes it even worse because it introduces more human error into the equation.

          The survey and methodology and polling firm are all suspicious. I just think there should be more evidence before saying results were in fact "faked".

          •  There IS strong evidence (2+ / 0-)
            Recommended by:
            BYw, Tm3

            The distribution of scores is strong evidence.

            Basically you only get the distribution they got if every student has the same aptitude for social studies, and is paying the same amount of attention.

            In America, 60% of bankruptcies are because of medical bills, and 80% of those people had health insurance

            by sullivanst on Thu Nov 05, 2009 at 12:34:55 PM PST

            [ Parent ]

          •  yeah, I'll stick to what I actually wrote (0+ / 0-)

            I agree that it's pretty much inconceivable that the reported results could occur in free response.

            I also point out that both the report on the OK study (linked above) and the longer report on the AZ study emphasize the importance of an open-ended format. To quote the OK story:

            In considering the profoundly awful results of this survey, it is important to bear in mind that an open-answer format represents a much higher standard than a multiple-choice-format exam, even with high-quality exams such as the NAEP.

            If the survey wasn't open-answer, that isn't so important to bear in mind!

            Then there's page 10 of the Goldwater Institute report, which lists ten different responses plus "don't know" for the first-president question. The report comments, "Some respondents gave some rather interesting answers, many missing by centuries."

            And I haven't even discussed the extreme weirdness of the results. Taking your example about political parties, how much sense does it make that given only two choices -- one with "Democrat" and one with "Communist" -- only 43% would get the question right? How much sense does it make that in a multiple-choice test, 0 Oklahomans would get at least eight out of ten questions right?

            I have a hard time making the IVR hypothesis work -- and I have a hard time exculpating Strategic Vision even if it actually conducted some research. But as far as I know, it hasn't been proven that Strategic Vision faked the results.

Subscribe or Donate to support Daily Kos.

Click here for the mobile view of the site