Extraordinary claims (are no big deal)
“Extraordinary claims require extraordinary evidence”. I don’t think there is a single sentence I dread reading more than this one when I receive reviews of a manuscript. Because this is not a statement about how science should be conducted or reported; it means “I, the reviewer, do not trust your paper, because it contradicts with my gut feeling about what is ordinary”.
Let me clarify two points (assuming that we live in an ideal world, in which data are not faked, claims in papers are based on facts and only on facts, and measurements are exact all the time; reviewers are still allowed to be disingenuous, for this is a true universal constant).
First, using this sentence in the review of an academic paper assumes that you agree to disregard its origin. Carl Sagan made this sentence — a bastardized version of Truzzi’s version, which focused on proofs instead of evidences— popular. The original context is not academic research; it is skepticism, and the claims of paranormal activities. Hardly the same.
Second, extraordinary claims, again within the context of academic papers, is a highly relative notion. When there is very little empirical evidence — or very little theory — , every claim is extraordinary (since there is, you know, no ordinary to begin with). Even when there are facts and theories (and that the two agree), the existence of an extraordinary fact (never mind that there is no quantitative definition), demands that an extraordinary claim be made.
Let me elaborate.
If there is an extraordinary fact — and it is assumed that there is no error in the measurements by which this fact is obtained — , it means that the current theory is not mature, or robust, enough, to explain it. All facts that are well predicted by the extant theory cannot possibly be extraordinary. In this situation, the only rational response is to make an extraordinary claim; namely, that something unanticipated happened, and the theory may be in need of revision.
And here is the issue — it all depends on what you assume to be true.
Reading “Extraordinary claims require extraordinary evidence” in a review often means that you claim something in contradiction with what the reviewer believes; and therefore, that your facts must be wrong. This triggers an arm race situation — you need to provide a quantity of fact that is larger than their belief. But one cannot possibly win; opinions are usually very robust against factual claims.
Writing that “Extraordinary claims require extraordinary evidence” in a review is nothing more than a way to disqualify a manuscript that surprises you. Don’t do it. Either the evidence is wrong — and then, whether it is extraordinary is irrelevant — , or it is not. Either it is well explained by the extant theory, or it is not. In the later situation, our gut feeling, intuition, are of little relevance.
There is, finally, very little chance that a single fact has enough significance to entirely dismantle an established theoretical framework, because we do not live in a Popperian world (and even then, it would be more complicated). In the contrary, a single fact, accompanied by its exceptional claim, will do more good than harm. Think of it as a sticky note — check why this system behaves this way. This will either lead to changes in the theory (unlikely), reveal that there was an issue with the evidence that has been missed the first time (there are well known precedents), or be classified as unproblematic background. The theoretical framework can only be strengthened by an extraordinary claim. So let us stop being so suspicious.