I've been promising for a long, long time to write a series of diaries here about how lay readers can read, evaluate, and even understand peer-reviewed science papers and third-party articles about them, even when they lack a background in the field in question. Unfortunately, real life has been doing the sorts of unpleasant things that real life does, and I haven't accomplished that. Or much of anything else here except drive-by commenting when I have a moment.
I still don't have that diary series started. But here, I'll continue the process of taking apart a really bad (pdf link) bit of science writing that was used as justification for extraordinary claims in a discuss a few days ago.
Many of the things that are wrong with this paper are wrong with the sorts of peer-reviewed papers used to by climate change denialists, the anti-vaccine movement, and no shortage of sham medical claims. Below the swirly thingy, we'll actually start looking past the title page.
Part I of this series looked at the paper's publisher, it's authors, and the organization they are all involved with.
Normally, one of the other things I start out doing with a paper I'm suspicious about is to hit the reference list, as was mentioned in the comments of Part I. Good papers, even when they're published under strange circumstances in bad journals, typically cite good papers. If what you're reading is full of citations to the Journal of Scientific Exploration (sort of the "premier" journal for fringe subject matter), odds are good that you're not really reading serious science. This paper has a bunch of citations to Russian material I'm not really familiar with. I could spend some time chasing them down to figure out what would be expected from Biological Series, Bulletin of Experimental Biology and Medicine, and Questions of Medical Chemistry, but I don't think that's a good use of time.
We already have reason to be a little suspicious of this paper's contents. Let's see what it actually has to offer.
The Big Questions
Especially when you aren't really familiar with the topic a paper is discussing and the field you're reading about, it helps to look at three big questions.
- What are the researchers trying to find out?
- What did the researchers do to find that out?
- What did the researchers actually find out?
In a nutshell, that's science. Sometimes a paper starts with a hypothesis, and tries to test whether it fits reality. Sometimes a paper starts with an observation, and tries to see how that might have come about. There are a lot of approaches to science and to doing science. Not everything works in quite the neat containerized approaches we learned from high school science fair projects. And medicine always complicates things, because pure experimental methods would be ethically abhorrent when you're dealing with real people.
But one way or another, you should be able to answer those three questions about any scientific paper. Even if you're starting from scratch on the topic and don't know what all the jargon refers to, and don't understand the underlying theories, you should at least be able to identify where the author answers these questions, and use that as a place to start your own further research if you desire.
Good papers make that process easy. Ideally, most of that is even in the abstract (that's the point, actually). This paper ... is not a good paper.
What are the researchers trying to find out
Well, this is about the only step where the page 1 "abstract" is helpful. The author is "studying of the state of cardiac activity in conditions of radioactive caesium influence." Okay. That seems like a reasonable premise for a paper. Obviously, the language is going to be a barrier here -- not from the jargon (although the paper's body doesn't spare that, either), but because whoever translated this from the original language was not particularly skilled. I'm not going to consider the possibility that this was originally in English, because my fragile little mind....
Anyway, that's a real question. What effect does radioactive cesium have on cardiac activity? So far, so good!
What did the researchers do to find that out?
Well, the abstract is no help. In fact, the abstract is less than
no help. See, reading that three-sentence abstract gives the impression that this is a review paper. A review paper doesn't report directly on experimental science. Instead, it gathers up all the results from a number of papers that were
about experimental science, and uses the miracles of statistics to combine their work to reach broader conclusions. These sorts of review studies are important, although really terrible things
happen when they go wrong (especially when they go intentionally wrong). Reviews tend to be a little harder to judge for a lay reader, too. Not only do you have to worry about how they selected their source material, but you have to be able to pass judgment on the statistics they used. Most of you can probably think of at least two or three things you'd rather do than analyze the integrity of a statistical comparison!
But when we skimmed through the paper back at the start of Part I, we saw a lot of experimental results, and not a lot of comparative statistics. In fact, there are three separate sets of experimental results here. None of them are clearly cited to previously-published sources, so I'm going to make the assumption that they're all new, and this isn't a review.
While the abstract isn't helpful, we do have a page-and-a-half forward. That's not the structure of most papers, but then, this was apparently published in book format rather than in a traditional journal. So we won't quibble too much. At least the author's trying to tell us what to expect, right?
Reading through the mangled English is slow going, but the point right now isn't to comprehend everything. It's just to answer one question: what science happened that is going to answer that question about radioactive cesium and the heart? After six or so paragraphs that basically try to tell us how the rest of science and medicine is failing to answer this important question, I decided to make like a kid reading a mystery novel, and skip to the end. The last bit of the forward has a numbered list that looks like it might be useful, but whose language is too strained to be much use. And then, at the very end, the author tells us.
[W]e undertook clinical, instrumental and laboratory investigations in children of different age, and also we carried out experiments with laboratory animals with the aim to study the effect of incorporated radioactive caesium on the state of cardiac activity.
There we go. There's going to be some sort of medical evaluation of children with radiocesium exposure, and some sort of lab animal study. We'll get into the details of those processes when we look at them in more detail later. But at least now we know what they did.
Also, we know it's weird. See, scientists like having lots of papers -- ideally, lots of good ones. That gives them more visibility, and more opportunities to be cited. Some positions even require a certain rate of publication. The push for a long publication record is one of the things that can get good researchers into trouble sometimes, and is one of many reasons why not every paper with a big name on the author line is a good one. Sometime there's even a joke in the scientific community about how much you can divvy up the research process into separate publications. "We can get another paper out of this!"
But here, it's not a joke. There's a human health survey and a laboratory animal study? Neither of which, as best as I can tell anyway, have been previously published? Those are two totally different types of science, requiring different overhead, different facilities, and (at least in the US and Europe) different levels of oversight by an ethical review board. And either of them alone would take quite a bit of time, which means quite a bit of money. Maybe we're just seeing it all here together because of the nonstandard publication format, but ... it's weird. And it's another reason to wonder just what's going on with this paper.
What did the researchers actually find out?
We'll skip past all the actual science for the moment. It'll be waiting for us. And with a paper that warrants as much suspicion as this one does, once we do start reading the experimental parts, it'll help us see if the author had to resort to any funny business to get from the data to his conclusion.
How does the conclusion start? Uh oh.
While hammering away the book, I was constantly thinking about
the necessity to bring to the notice of any civilized person the information
about harmful influence of radioactive substances when they incorporated
by the organism. Unfortunately in society there is, at best, indifferent
attitude to that problem. That is why we pay a very high price, which is
human’s life. Intellectual dark turns about the tragedy. To a great extent
the blame rests with medical scientists. They don’t only try to inform
population, using earlier obtained data, but don’t even study alterations,
appearing in the organism when radioisotopes incorporate.
Curse those medical scientists! Why don't they warn people? Why don't they even study
the dangers that I'm studying?! Civilization -- society -- must read my book!!
Well okay then.
Sometimes science discovers that something we've been doing for a long time is actually very, very bad. Leaded gasoline. Ozone-depleting CFCs. Global climate change. Or, on a smaller scale, any number of studies that revealed that FDA-approved drugs turned out not to be as safe as we thought they were when they got approval. All these sorts of things have to overcome existing consensus. That's an ugly process, sometimes. DuPont's chairman famously called the ozone depletion theory "a science fiction tale...a load of rubbish...utter nonsense." And everyone here should be familiar with the battle over global warming.
When you think you've found something that bucks conventional wisdom, there's a right way and a wrong way to point that out. The right way is to, well, present your science like a scientist. Point out that, yes, there's been a lot of work putting into showing this, but you've got some really good data that says that. The wrong way, generally speaking, is to rant about how everyone else is ignoring the Truth and is happy to just let people die. This doesn't foreclose the possibility that there's real science here, but it's probably the biggest black mark so far. A lot of anti-science authors, especially in medicine, take this attitude towards presenting their findings. The anti-vaccine movement is pretty much built of the stuff.
So, with that aside, what is the actual conclusion of the paper?
Its [radiocesium] penetration in myocardium cells causes structural and metabolic alterations, leading to the energetic deficit and to the impairment of their
functions and in some cases to death.
Or, in more readable English -- radiocesium penetrates into heart muscles, causing structural and metabolic changes, leading to functional impairment and, in some cases, to death. Yikes! That's a pretty grim conclusion. When we start digging into the experiments, we'll see whether the science gets us there. But that will have to wait until later in the series. Before we call this a diary, though, there's one last little problem hidden in the conclusion...
It should be noted that the effect of incorporated radioactive caesium on human’s and animals’ organism assumes its participation in plastic and metabolic processes and first of all like chemical element and not as a source of radiation.
Um, wait. What? Remember the first of our three big questions. This paper is supposed to be studying the effects of radiocesium. That probably refers to Cs-137, which is a major components of the radiation burden after nuclear accidents like Chernobyl and Fukushima (spoiler: it does). But here, at the very end, the author "assumes" the effects are just from its chemical interactions rather than from radiation? So ... is the paper really just looking at the cardiac effects of cesium in general? What's going on here?
When we dig in deeper, we're going to have to look for an answer to that question.