Skip to main content

View Diary: How to evaluate the quality of scientific research (136 comments)

Comment Preferences

  •  I wouldn't necessarily agree (14+ / 0-)

    that secondary sources are better than primary sources. Often secondary sources miss key points that the primary source was making about their data. They can also repeat misinterpretations of the primary data. As a working scientist, there have been a number of occasions where we've seen a review article make a statement about a primary source's data and then when we went and checked the original primary study we found that the review article was completely wrong about what the primary source was saying.

    And the reliance upon high impact journals is often misplaced. There have been many occasions where a study was published in journals Science or Nature (generally considered at or near the top of science journals) only to be later retracted for some error that a reviewer completely missed.  And just because a paper from one of those journals was not retracted doesn't mean it didn't have significant errors either. I'm not saying impact factor isn't important, but it's import is often over-stated.

    •  Still, a good review can put primary sources (1+ / 0-)
      Recommended by:

      in context, and provide an overall understanding of an area.

      •  That is definitely true. (2+ / 0-)
        Recommended by:
        tarkangi, Mike Kahlow

        That's really the purpose of a review - put all the primary data into a bigger picture.  Our lab has been involved in writing several reviews for our area of interest and that is always what we try to do.

        •  Good for you (2+ / 0-)
          Recommended by:
          madmsf, FindingMyVoice

          The right thing to do, always, is to go to the primary literature and read all the relevant papers and make certain that you personally understand all of the arguments and all of the data and that you can see directly how the data supports (or not) the alleged conclusions.

          The trouble with this is two fold: it takes a lot of time, and most people are simply not competent to make such determinations on their own.  It doesn't matter how much experience you have on Google, without the proper education and experience in following arguments and evaluating evidence you will be a babe in the woods regarding the primary literature.

          So good quality reviews by primary researchers are a great stand in.  They bring in the relevant evidence - excluding the irrelevant - and tie it together in to the generally accepted conclusions.  This doesn't guarantee that it's always right, but it's certainly the most reliable that we have in light of present understanding and it is certainly a dependable guide on the way to bet.

          Much lower in priority are the press releases, which invariably have some kind of ax to grind, and propaganda sheets from fixed ideology web sites are the absolute worst places to get information.  They depend on people being ignorant and fearful, and will keep peddling their nonsense despite repeated corrections.

          Vai o tatu-bola escamoso encontrar-me onde estou escondendo? Lembro-me do caminho de ouro, uma pinga de mel, meu amado Parati (-8.75,-8.36)

          by tarkangi on Thu Jul 17, 2014 at 06:47:46 PM PDT

          [ Parent ]

    •  And major pubs have issues IMO (0+ / 0-)
      And the reliance upon high impact journals is often misplaced.
      And I think it is a conflict of interest that many journals, such as the stable of Journal of the American Medical Association publications run advertising.

      "Societies strain harder and harder to sustain the decadent opulence of the ruling class, even as it destroys the foundations of productivity and wealth." — Chris Hedges

      by Crider on Thu Jul 17, 2014 at 01:57:58 PM PDT

      [ Parent ]

      •  How do you suggest they be funded? By whom? (0+ / 0-)

        I'm not completely criticizing your point, but it's not cheap to run a journal. I suppose the broadest range of funding from lots of sources, commercial and not, would be best if we had a perfect world. I don't know, what percentage of their funding comes from advertisements?

        A good journal or newspaper or whatever will not let their advertisers get in the way of accurate reporting even if it hurts the advertiser's reputation. If they don't like it they can take their business elsewhere.

    •  yup (4+ / 0-)

      No substitute for actually reading the study.  I have discovered on more than one occasion the data actually says something different than the authors claim.  And such claims are often repeated uncritically.

    •  absolutely (0+ / 0-)

      I've personally written at least 4 Comments on Phys Rev Lett papers, i.e. caught 4 seriously wrong papers in a highly prestigious journal. I knew of one paper in Nature in which people in the lab said the data were more or less faked. The diarist's obsession with impact factors is entirely misplaced.

      Michael Weissman UID 197542

      by docmidwest on Fri Jul 18, 2014 at 05:52:13 AM PDT

      [ Parent ]

      •  Tentatively agree. (0+ / 0-)

        Stuff that's wrong and/or incomplete can make it even into the most prestigious journals. The review process is only as good as the people involved, and mistakes can be made.

        That said, it's probably a good way to weed out crap if the only place you can find evidence is in some pay-to-publish junk journal.

Subscribe or Donate to support Daily Kos.

Click here for the mobile view of the site