I very rarely write on DailyKos, but after seeing the diary by skralyx about the recent “Sodom” archaeology paper on the rec list for so long, and particularly the comments in Nowhere Man’s diary refuting the same paper, I felt the need to wade in. While this community is generally supportive of science, it’s clear that many people don’t have a good understanding of how science is conducted on a day-to-day basis, particularly “historical” sciences such as archaeology.
This is a personal narrative, so there won’t be a lot of links, but there will be anecdotes. Take that as you will. I want to stay pseudonymous, and I’m not going to out myself for the sake of this debate. Some of you may figure out who I am anyway; if so, I ask that you not make it public.
So, some background to establish my credentials. I am not an archaeologist, but rather a paleontologist (and no, they aren’t the same thing!), with more than 30 years of lab and field experience. To date, I’ve published 26 peer-reviewed papers with the 27th currently in review, and have been involved in naming 5 new species of fossil animals (3 as first or sole author). I have also been a peer reviewer for 10 different journals and editor for one journal. Now that I’m a more “senior researcher”, instead of running a lab I’m director of a museum that holds more than 1.5 million fossils and artifacts (many held in trust for the US government), and I supervise the work of several paleontologist and archaeologists who work at my museum.
Peer Review
One of the things that sets scientific work apart from most other fields is peer review. Here’s how that process works, although the details vary from journal to journal:
When you feel your paper is complete and ready to publish, you submit it to a journal for consideration, where it goes to an editor. The editor may reject the paper immediately (a “bench rejection”) for a variety of reasons (not appropriate for the particular journal, for example). Or the editor may send the paper for review.
The editors will select 1-3 volunteer reviewers (number varies by journal), ideally experts in the field the paper addresses. Some journals allow authors to recommend possible reviewers, or to “blacklist” certain reviewers, but the journal is under no obligation to follow these recommendations. (As an aside, why would you blacklist someone? Well, scientists are people too. Some people may be destructive reviewers, that try to sink a paper instead of make it better, or they might be unethical and steal the info gained during a review, or try to damage the author’s reputation, etc. Some people, including some scientists, are just jerks.) At reputable journals, the reviewers generally must not have collaborated with the authors for, say, the last five years, and can’t be from the author’s institution.
Each reviewer will make their comments and recommend to the editor whether or not the paper should be published. The editor may then accept the paper as-is (this almost never happens), provisionally accept the paper with minor or major revisions, or reject the paper. Rarely, with large, complex, or controversial papers, the editor may first send it to additional reviewers before making a decision.
If your paper is rejected, you can choose to submit it to a different journal. If it’s accepted with revisions, you can make these changes or, if you disagree with the reviewer’s changes, you can explain to the editor why you think the reviewer is wrong. Once the editor reviews your changes and responses, they can accept it, send it back to the original reviewers or to new reviewers, or reject it. The whole process can take anywhere from a couple of months to a year or more.
So, rigorous peer review, if done correctly, means that 2-5 other people have seen the paper and had a chance to suggest revisions. That’s good, but it’s not actually a lot of people. Most papers have multiple authors, each with their own expertise that contributes to the work. In my field of paleontology, a paper may include a morphologist that specializes in the study group, a phylogenetics specialist (that works on species relationships), a statistics expert, a stratigrapher, and some combination of dating experts, functional morphologists, taphonomists, etc. You’re not going to get enough reviewers to cover all of those specialities, so in a sense peer review is just a spot check; “At first glance, there are no glaring problems”.
This means that, even if everyone has the best intentions, incorrect conclusions and even shoddy science get through peer review ALL THE TIME. A completely innocent personal example:
I once thought I had found a fossil of a particular group of mammals from a time and place they had never been found before. I wasn’t completely sure, so I consulted with a colleague who had published on that group, and he thought my interpretation was good. I spent several months examining the available literature and found nothing to refute my interpretation, so I presented the preliminary idea at a professional meeting, and got universally positive feedback. So I submitted to a journal; one of my reviewers was the world’s foremost expert on this group. He gave me an excellent review, and even suggested some just-published research that indirectly supported my conclusions. The paper was published with revisions. About 9 months later I was sent an email by an avocational paleontologist complimenting me on the paper, and incidentally commenting on how much my specimen looked like a bone from a previously published fish. I was unfamiliar with the fish, but when I searched for images of it, sure enough I confirmed that my supposed mammal fossil was actually a fish! Keep in mind that more than a dozen paleontologist saw this work before publication, including the top two world experts on the particular group, and still my erroneous identification was published. (I’ve since published a correction.)
Peer review is a great first step in vetting science, but it’s not the last word.
One other note; Scientific Reports is not actually a highly regarded journal, especially in the historical sciences, largely because of rather shoddy peer review. While it was designed to compete with journals such as PLOS ONE, it is not actually in the same league in terms of quality or reputation.
Science Twitter
Twitter is an absolutely horrible place, even without Donald Trump. But there are corners of Twitter that have their own culture and momentum. Many scientists have embraced Twitter as a tool to communicate both with the public and with other scientists. I’ve personally first met some of my eventual collaborators on Twitter, and even had research projects that began as Twitter threads (or as blog posts, in the old days). These days it is common for authors, upon publishing a paper, to both publicize and describe their research on Twitter. This allows scientists to directly communicate their research to the public. It’s also common for other scientists to describe newly published papers from other workers, which might include criticizing it; in fact, Twitter is now often the first place where new discoveries get their first post-peer-review assessment. This is normal and appropriate. Remember that peer review is the MINIMUM vetting a paper should receive; once it’s published is when the real assessment begins, when all scientists (not just a handful of reviewers) have had a chance to see it. If your research is controversial, and you haven’t covered all your bases, you better buckle up, because the amount of scientific expertise on Twitter is FAR beyond anything you’ve experienced in peer review.
So far, the Tall el-Hammam paper is not standing up very well to the increased scrutiny.
Biases
Scientists are people. As people, we all have both conscious and unconscious biases that can easily adversely affect our research. Part of what future scientists should be learning in graduate school is how to recognize biases in ourselves and others. It’s not easy to recognize your own biases! That’s one of the reasons we have peer review, and it’s why most scientists run their work past multiple colleagues before submitting it. I’ve been fortunate to usually work in places that had at least one other paleontologist, and that my wife is also a scientist, because it has given me a way to test my ideas at early stages. I’ve had lots of hypotheses that never made it to the writing stage because of an overlooked angle or an unrecognized personal bias. You really have an obligation to explore your own potential biases. Another personal anecdote:
A couple of years after taking a new job, quite by accident, I thought I had discovered evidence for a new species in my museum’s collection. But this wasn’t just any new species; it was a member of a group that was well-known to the public, and specimens of this potential new species had been on display (under a different name) at several internationally-known museums for decades. Publishing this species would be a major score for my museum. This introduced a potentially powerful bias and conflict of interest; it looked very “convenient” that just after starting work at this museum, I happened to find a species that had been overlooked at other museums for decades, especially since it was a group for which I had no publication record. If we published this paper, I knew that readers might be suspicious of our conclusions because of this, and so our work would have to be iron-clad. It also made me doubt my own interpretation, and I spent more than a year testing and retesting my conclusions, collecting additional data specifically to try and disprove my hypothesis, and discussing my ideas with colleagues. I invited several coauthors with different specialties to join the project. One person, who had a lot of experience working on this group, actually turned down joining the paper, because she didn’t agree that we actually had a new species.
In the end, we finally decided to submit. I requested that one of the reviewers be the same person that turned down my request to be a coauthor, knowing that she disagreed with us and would give a fair but critical review. She ended up giving us an excellent review, telling the editor that she had been skeptical but was convinced by our arguments. Our other reviews were also good, we made our revisions and the paper was published. It rapidly received positive feedback on Twitter and other places, and our new species is now widely accepted in the field.
In short, potential biases of the authors are absolutely fair game for criticism. If the research is legitimate, and the author has potential biases or conflicts of interest, it’s incumbent on the author to attempt to assuage doubts about their biases by making their arguments as strong as possible.
The Tall el-Hammam authors have a ton of reasons to suspect their motivations. Some of the authors have a history of claiming evidence for impacts that has not stood up to detailed scrutiny. Others are affiliated with organizations that have the explicit goal of finding confirmation of biblical stories. There’s nothing automatically wrong with that goal, but if you claim to succeed, it invites extra scrutiny.
Pseudoscience dog-whistles
This section is a bit more confrontational. Several commentators suggested that the Tell el-Hammam authors didn’t explicitly claim that their site was Sodom, and that therefore attacks on the paper were hysterical Bible-bashing. In fact, the authors do imply this possibility, although it’s not a central theme of the paper. But this approach is something that I’m familiar with as a paleontologist, as it is used extensively by “creation scientists”. Creationists realized a long time ago that if they openly stated their goals, journals would reject their papers, and courts would rule against them on 1st Amendment grounds, Thus the shift to concepts like Intelligent Design (ID). If someone accused an ID paper as being a creation work, the authors would immediately point out that they never said “creationism” or even “God” in their paper. But the dog whistle is perfectly clear. I’m not an archaeologist, so I don’t follow the field that closely, but this sure sounds to me like the exact same scenario - especially given the publication histories and potential biases of the Tell el-Hammam authors. They may not explicitly talk about Sodom (much), but everyone knows that’s exactly what they mean.
Again, I’m not an expert in any aspect of this paper. But, given my experiences in paleontology, working with archaeologists, and serving as a peer reviewer and an editor of numerous papers, this work has a ton of red flags. If the paper isn’t retracted, there may well be a formal response in the literature in the coming months, and it won’t be pretty. In fact, it will probably look a lot like the Twitter threads that are out there now.
And one final note: I think there was no problem with the initial diary posted by skralyx. It was a good description of the claims the authors made, and led to some good discussion. It was the attacks in the comments of that diary and Nowhere Man’s that led me to write this.
Please keep it civil in the comments!