I've been promising for a long, long time to write a series of diaries here about how lay readers can read, evaluate, and even understand peer-reviewed science papers and third-party articles about them, even when they lack a background in the field in question. Unfortunately, real life has been doing the sorts of unpleasant things that real life does, and I haven't accomplished that. Or much of anything else here except drive-by commenting when I have a moment.
I still don't have that diary series started. But here's something to tide you all over: an exploration into a really bad piece of science literature. In fact, it's such a bad piece of science literature, it's going to take me more than one diary to disassemble it for you!
Where'd it come from? I've been involved in a running dialogue/discussion/argument in the comments of a days-old diary. I'm not going to point to it, because I don't want to call out the commenter with whom I've been interacting. If you've got to know, it's not rocket science to find it. The important thing is that the commenter was making a claim about the health effects of really, really small amounts (on the order of picograms) of the radioactive isotope cesium-137. I considered those to be extraordinary claims and demanded a reference for them.
I got this (pdf link). It's one of the worst papers I've seen in a long time that wasn't about something that couldn't automatically be branded as hoakum. And now, you get to explore it with me!
Below the swirly thingy, we'll get started with a quick skim of the paper for a general impression and then get into the swing of things ... with the title page.
First impressions
Okay. So you're in a debate on a contentious science topic -- in this case, the effects of nuclear fission byproducts on health -- and your opponent links you to a paper defending their side of the argument. Awesome! That's better than you can expect from 75% of the Internet. You're on the right track! And, holy crap, it's not even paywalled. Score.
So the first thing you should do is just sort of flip through it. You're not reading for comprehension right now. You just want to get a sense of the thing. Some of these papers have the lay reader in mind, and spell everything out carefully. Some of them ... well, aren't. Before you start asking hard questions about peer-reviewed science, it's nice to know what you're getting in to. So what does this paper have in store for us?
Yikes.
It's rich with high-end medical terminology, none of which is glossed. And worse, it reads like it was mechanically translated from Russian, which it might very well have been! "Being the leading in the organism, cardio-vascular system is very sensitive to the influence of different environmental factors." That's what we start out with. Okay then, this is not going to be easy going. But we'll give them the benefit of the doubt. No matter what some folks think, the whole world doesn't speak English. At least these folks gave us an English-language version!
Where, and by Whom
Not all papers are created equal. Once you've cracked your knuckles and decided to look at the meat of the issue, always start with the journal and the authors. There are a few flagship journals in each field, but competition for space in them is steep. It's not bad science just because it's in a B or C list journal. On the other hand, sadly, there aren't any restrictions on who can call their web-based junk mailer a "journal", so everyone and their homeopath can claim to be published in one. If you're handed something from the Antarctica Journal of Mathematics, you can probably quit right there (if the name didn't clue you in that that one's a scam journal, the web design will).
So what do we have with this paper? Uh oh. This isn't a journal publication at all. "This book is one from the series of the author’s publications...." That's not really a good sign. We'll note that the publisher is listed, though. Something called "Belrad". We'll get back to that. But it does at least claim to be peer-reviewed, right...? Sort of. Real peer-review, when it works, is a double-blind process. The author doesn't know who reviews his paper, and the reviewers don't generally get to know who the author is. This isn't like that. We've got two named, and presumably hand-picked, reviewers. That means, at best, they're more like editors.
So, the author. Singular. Hmm. Modern science, especially tricky stuff like medicine and nuclear science, is complicated. There are almost always multiple authors. See, everyone who takes part in doing the science wants their name on the list; that way, other people know they've been doing stuff. It's how you build a resume in published science. This paper? Just one guy. "Here the author analyzes results of his own clinical and laboratory researchers...." I presume that means "laboratory research", because if there were other researchers involved, he should have named them. I'll provide a bit of spoiler: there's never any indication that anyone else assisted in the preparation of the studies described here. That's okay in a review article that just takes a bunch of other papers and combines their results, but this is primary research. Primary research that involves nuclear science and cardiac medicine -- two fields that do have some overlap, but aren't exactly shared backgrounds. Does the single author listing mean he's a glory hog who wouldn't let his associates share credit? Or does it mean he really did all this alone? Either one would be cause for concern.
So who is he? As it happens, Yury Bandazhevsky has a Wikipedia page. That's not the best way to do research into people, because it's fairly easy to pad the Wikipedia entry of obscure scientists. But it's a place to start. So he got his degree at ... some institute in Belarus without a Wikipedia page? That's not a good sign. And he spent time in prison for takes bribes from students' parents, in what has been claimed to be political persecution for his controversial scientific stance about Chernobyl? Hrm. We don't have enough evidence to take a side on the political issue, but that's enough to make us cautious. Regardless of anything else, this author is likely to have an agenda. And that goes double because he's going it alone. Confirmation bias is an easy trap to fall into for all of us, scientists included!
How about those reviewers? Were they likely to give an unbiased review of this work? The first is Michel Fernex. He's part of an anti-nuclear NGO. He's president of something called "Children of Chernobyl Belarus" and he wants to separate the WHO from the IAEA? That sounds like there might be a question about whether he'd give an unbiased analysis. And his qualifications seem thin: he's cited as an "honorary professor". And the other one, Vassili Nesterenko? Well, his qualifications are at least a little better. But he has a deeply personal involvement with the Chernobyl incident, and he's another "persecuted scientist" with a complex political history (assassination attempts?). These reviewers clearly are not serving the role of traditional peer review. We should proceed with caution.
As part of this step, it's also nice if you can figure out who funded the research in the paper. Good scientists come out and say it, along with declaring any conflicts of interest. It's not necessarily a bad thing when a study is funded by, say, a corporation in the field. We all know about the cigarette industry's propped-up research. But in some fields, the corporations are the only people with the money to fund things. You have to take what you can get. This paper doesn't really come out and say who funded things, but it's not hard to figure it out. Remember the publisher of the paper? Belrad? Wikipedia helpfully tells us that our reviewer Nesterenko is director of the Belarusian Independent Institute of "Belrad". And hidden in the paper's text, Bandazhevsky lets on that they're providing at least some of his equipment. That's the sort of conflict of interest it would have been nice to provide up front.
What kind of place is Belrad? Well, they don't seem to have a Wikipedia entry, but not all laboratories and independent research groups will. They do have a webpage, which is nice, and it even comes in English (well, more or less). They're dedicated to monitoring cesium-137 accumulation in Belarus, and its effects on local children. Wow, that's sort of specific. And it's exactly what this paper is about. That's a little too convenient, really. What's worse, they produce and sell a product, "Vitapect", designed to combat those health effects. Like I said before, it's not always bad when the people funding science have a financial stake in the outcome looking a certain way, but this all has such a narrow focus that it raises serious doubts.
That's not all the issues you could take home from their website, but for now, that's enough. Even before starting to read the paper itself, we've got a lot of warning flags. In the next installment, we'll get into the science.
