Yes, you read that right. This so-called shill for the pharmaceutical industry is calling them as I see them–most published medical research articles present information that is eventually a failure.
Now by failure I mean that more often than not, claims that are supported by one or two published articles, rarely lead to a marketed product. I don't mean that the research is fraudulent, although some are, especially in low level journals frequented by pseudoscience pushers. I don't mean it's bad science, although there's evidence of that, which I'll discuss below.
And I don't mean that there's some grand conspiracy between Big Pharma and everyone else (again, no evidence to support that ignorant nonsense), although there is evidence that some research sponsored by Big Pharma is poorly done.
So what do I mean? Results from lot of medical research that get splashed in the news rarely, and I mean rarely, ends up having any clinical utility.
This does not mean that procedures, pharmaceuticals and devices that have been vetted through lots of research that repeat and confirm the original data and that form the basis of a scientific consensus are bad research. Almost everything passed by the FDA and other regulatory agencies in other countries meets high standards for risk and benefit analysis.
Finally, arriving at a scientific consensus is a brutal, time-consuming process. It means that the theory or idea has been repeated many times, and the analysis and data are solid. So even though "most" research ends up in a failure, that's because science is harsh to research that can't be repeated, or was badly designed.
The best research isn't a failure, even if it finds negative results. And the best ideas in medicine, let's say vaccines, have been so thoroughly vetted that the consensus is nearly unassailable. Though people try with their poorly designed, unrepeatable research.
Medical research to clinical utility
In 2003, researchers reviewed 101 studies published in respected, high impact factor scientific and medical journals between 1979 and 1983 that claimed a new therapy or medical technology was clinically promising. What was shocking was that they found only five, of those 101, made it to market within a decade of publication.
Why did they choose a decade? That's a good period of time for further research, clinical trials, and finally FDA approval.
So that's just 5%.. But what's even more ironic is that only one, ACE inhibitors which are used to treat hypertension, was still used at the time of publication (2003). Of course, it's still used today as one of the basic tools in treating high blood pressure.
Think about that. There were 101 breathless claims of some miraculous medical breakthrough (if the PR departments of the various universities and startup companies did their jobs), and only one is on the market today.
Cancer research to clinical utility
I often make a pointed joke about research into cancer. "We have cured cancer in mice over the past decade." Setting aside the fact that there will never be "one cure to cure them all," given that there are over 200 cancers, each so different that it would be impossible to imagine one drug.
But when you read an article that eating blueberries will prevent breast cancer based on one mouse study, you need to consider it very skeptically. The gold standard of medical research is a double blind clinical trial, repeated several times, and then rolled up into a systematic review–the basis for a scientific consensus. At that point, if it showed significant clinical differences, I'll be eating blueberries 24/7.
The marijuana activists love to tell you that cannabis can treat cancer. Yet, the only evidence of its ability to treat cancer is in one poorly designed clinical trial for one specific cancer, and a few animal studies. Of course, the level of the active ingredient that might attack some cancers would require a blood level of THC that would kill the smoker. Well, one couldn't smoke the hundreds of joints necessary to kill this cancer–a rate that would have to be sustained every day until the cancer was "cured."
Matthew Herper, writing for Forbes, talked with one pharmaceutical industry consultant that there were over 200 failures of cancer treatment drugs over the past few years. Failure in cancer treatment is much more common than success. And this is why accepting the hype of a treatment that is successful in mice and rats, or in one or just a handful of real patients, is going to lead you down the wrong path of cancer treatment.
There are so many drugs that appear to work with one person, but that could be just random. Sometimes people get better from cancers because of unknown reasons (and don't take that to mean that obviously it's blueberries, it's because sometimes the body gets it right, for reasons we will understand eventually). So if you base your hype on ONE study with ONE patient, you will be mislead.
But as Herper's article says,
There are two reasons to be wildly optimistic about the state of cancer research. One is that new treatments that harness the immune system to attack tumors do lead, in some cases, to long-lasting remissions, even cures. This includes the treatments Vice profiles, but also drugs that make tumor cells visible to our bodies so they can be destroyed.
The second is that new genetic technologies are allowing doctors to start to look at tumors based on what mutations have caused the cells in them to go haywire. That means targeted drugs like Gleevec, Rituxan, and Herceptin that add years to patients’ lives. It may also mean new types of diagnostic tests that can detect cancer early, when genetic material from tumors first starts to circulate in the blood. I think we truly are at a moment of revolution.
This is truly a wonderful technological advance in treating cancers. But, we've heard this before. Remember, there were 101 medical advances that were hyped just like this, and only one ended up having any clinical relevance today.
One more point. Over 90% of cancer drugs that enter Phase 1 trials ever get FDA approval to be marketed (pdf). It more or less debunks that myth that the FDA is in the pocket of Big Pharma. Or someone needs to up the payments or something, because a 10% success rate probably indicates that the FDA is pretty tough on these applications
Evaluating research
Every day, new research crosses my desk, and I try to evaluate it quickly, because I just don't have the time.
So here is research in order of highest quality, that which provides the basis of a scientific consensus down to garbage, useful to no one except those who deny scientific reality.
- Secondary reviews published in peer-reviewed, high-impact journals. These secondary research articles include meta-reviews, review articles, and Cochrane Collaboration reviews. These studies essentially roll up the data from possibly dozens of other research articles, while attempting to remove obviously poor quality research or biased data. They are mostly useful for examining numerous randomized clinical trials, providing the reader with a higher number of data points, usually with better statistical analysis.
- High quality randomized controlled trials with definitive results. These are studies that include fairly large numbers (in general, I like to see >1,000 subjects in each arm), with confidence intervals (errors) that do not overlap, and show a clinically significant effect. And published in high impact journals.
- Randomized controlled trials with non-definitive results. These are small trials, usually with large confidence intervals which only suggests a clinically significant effect. At some point, these type of studies border on the level observational studies that need to be confirmed (or refuted) by larger studies.
- Cohort studies (retrospective studies). These are an analysis of risk factors and follows a group of people who do not have the disease, and uses correlations to determine the absolute risk of the subjects contracting a disease. A cohort study is often undertaken to obtain evidence to try to refute the existence of a suspected association between cause and effect. This is why we know that vaccines do not cause autism.
- Case-control studies. This is a type of analytical study which compares individuals who have a specific disease ("cases") with a group of individuals without the disease ("controls"). The proportion of each group having a history of a particular exposure or characteristic of interest is then compared. An association between the hypothesized exposure and the disease being studied will be reflected in a greater proportion of the cases being exposed
- Cross sectional surveys. Essentially, these are surveys where the research observes one single point of data in a large population. It provides little information about correlation or causality, but is used to determine what proportion of an observed population may have a disease at a single
- Case reports. I dislike these types of articles. They often show up in high quality medical journals, but they are usually observations of one or two patients. Those in medicine know their purpose, which is to give a "heads up" to an observation. They have no scientific validity beyond observational. Unfortunately, science deniers will pull a case report published, and use it to condemn a whole field of medicine. Don't use them to support your argument, one way or another.
- Animal or cell culture studies. This is basic primary research which may or may not be applicable to human biomedical science. Too much credence is given to studies that may be 20 years away from having any applicability, if any at all. The National Cancer Institute has screened over 400,000 compounds for treating cancer, and maybe 20-30,000 have even made it to early clinical trials, and of those just a handful are used in modern medicine. You have to be extremely skeptical of reading an article that has source to a press release that might overstate the results (or even if they refer directly to such a primary study).
- Meeting abstract or poster presentation. These are presentations made at meetings/conventions of a scientific society. There are hundreds made every year, and preliminary research is presented in poster sessions or formal presentations. They are not published formally through a peer-review process, and they often do not contain enough explanation of methods, data, analysis, and other issues to evaluate properly. In an article in JAMA, the authors found that within 3 years after abstracts were presented at meetings, 50% were published in high-impact journals, 25% in low-impact journals, and 25% remained unpublished. Interestingly, the 39 abstracts that received front-page coverage in newspapers had a publication rate almost identical to the overall publication rate. The authors concluded that, "abstracts at scientific meetings receive substantial attention in the high-profile media. A substantial number of the studies remain unpublished, precluding evaluation in the scientific community."
- Press releases or news reports. Do not accept the conclusions stated by a press release from a major research university, they haven't been peer reviewed. I have a feed filled with press releases from major research universities, and I've found errors in interpretation from the university's public relations office relative to the real research. However, it is possible to use a press release to chase down the actual published, and peer reviewed study. So, it's not completely worthless.
- Natural News or Mike Adams, the Health Ranger. Let me make this clear, Natural News is a foul, fetid, putrid sewer of information, which befouls any real science with lies. The antisemitic, anti-science, anti-knowledge website, Whale.to, similarly reeks of malodorous smells of pure garbage. Anyone who uses either site as a source for anything in medicine loses the discussion without any further debate.
What does this all mean, the TL;DR version?
- Be skeptical of any medical research until it exceeds the standard of double blind clinical trials, is repeated by other researchers, and a meta review of all the results supports the hypothesis.
- If the whole basis of your medical claim is a poorly designed study, mice, or published in a bad journal, it should be considered, but ultimately shouldn't have any influence over real clinical decisions.
- The FDA and Big Pharma are not in any kind of conspiracy given that most drugs fail to get approved. This is one of the most laughable conspiracies ever.
- Cherry picking research to support your a priori beliefs? That's really not going to work
Key citations: