Skip to main content

View Diary: Researchers finally replicated Reinhart-Rogoff, and there are serious problems. (124 comments)

Comment Preferences

  •  "Reproducible research" - the gold standard. (1+ / 0-)
    Recommended by:

    This is a glaring example of why we need to make "reproducible research" the gold standard for any scientific publication. That means that the authors must be prepared to document all steps in their paper, from raw data to published numbers and graphics, in such a way that a knowledgeable person could replicate the findings.

    You need to define, in a reproducible way, the inclusion and exclusion criteria (for people, countries, years, whatever.) You need to say what variables where chosen and how they were modified, and how they were analyzed. You need to be able to produce the programming code - R, SAS, or whatever.

    Excel spreadsheets fail this standard. Period.

    This is the sort of mess that got a Duke cancer researcher fired (though of course he also claimed to be a Rhodes scholar, evidently forgetting that such a claim could be checked.) You can read the sordid details on that one if you google "Potti".

    I would like to see this receive the same sort of "60 Minutes" treatment that the Duke debacle got. And I'd like the economics world to be scared enough of the consequences to say, as some of my colleagues do, "Please do this super carefully and take whatever and resources you need. I don't want my work to show up on 60 Minutes."

    •  science is the scientific method (0+ / 0-)

      which means putting theories to empirical test.

      Science isn't math, or published papers. It's putting predictions to the test.

      •  Well, that's why economics isn't quite science. (0+ / 0-)

        Economics is math, trying to describe the real world. Economists don't have big model economies where they can flip the AUSTERITY lever on one and leave the control group alone. The best they can do is reverse-engineer some equations from observations of nature.

        But even if your “experiments” consist only of crunching numbers, they should of course be reproducible.

        Code Monkey like freedom / Code Monkey like peace and justice too
        Code Monkey very nerdy man / With big warm fuzzy bleeding heart
        Code Monkey like you!

        Formerly known as Jyrinx.

        by Code Monkey on Wed Apr 17, 2013 at 06:46:26 AM PDT

        [ Parent ]

      •  Empirical tests require reproducible research. (0+ / 0-)

        As a scientist with more than 40 years of research experience, I am well aware of the importance of scientific methodology. But the scientific method requires more than just careful study design. It also requires making sense of the raw results in ways that do not introduce bias or error, and can be validated by other researchers. Hence the importance of math, and of published papers. How else are we to determine whether the findings of the "empirical test" will stand up to scrutiny?

        The gold standard in my world is randomized clinical trials, preferably double-blind, or at least blinded to the evaluators. But even careful empirical testing is worthless if the pathway from raw data to published paper cannot be validated.

        Some problems and questions are not amenable to randomized testing, for example, assessing the effects of potentially toxic and non-beneficial substances in human beings (eg cigarette smoke.) Observational studies pose challenges to researchers, but it is still possible to pose well-thought-out predictions based on theories, and to examine them in studies where evidence could contradict the prediction and thus cast doubt on the theory.

        So you can develop predictions and put them to the test even in, say, epidemiology, which has developed careful methodology for observational studies, or economics. Well-posed questions and careful definition of scientific tests will only take you part-way to believable evidence. You also must have reproducible research.

Subscribe or Donate to support Daily Kos.

  • Recommended (125)
  • Community (63)
  • Elections (24)
  • Media (23)
  • Environment (23)
  • Culture (22)
  • Civil Rights (22)
  • Science (21)
  • Law (21)
  • Trans-Pacific Partnership (21)
  • Josh Duggar (20)
  • Labor (18)
  • Economy (17)
  • Ireland (16)
  • Bernie Sanders (16)
  • Marriage Equality (16)
  • 2016 (15)
  • Hillary Clinton (15)
  • Climate Change (15)
  • Health Care (14)
  • Click here for the mobile view of the site