Thanks for the comment. I agree a core problem is that we are trained, and that we train our students, to make, publish, and defend strong claims based on single studies. The more surprising the result is, the greater the reward. And as noted by Ulrich Dirnagl in his blog post, the OPERA experiment was repeated several times yielding the same surprising results, nicely demonstrating that also “replication of an experiment in the same laboratory is of very limited value.” I thus agree that reliable conclusions can only be drawn using cumulative evidence from multiple independent studies.

However, independent replications are rarely done and may often not appear necessary if the result of a first study comes in as non-surprising and as meeting the expectations. Jose Perezgonzalez writes me via Email: “Had the results been that neutrinos travelled slightly below the speed of light, nobody would have actively checked for errors, despite the loose cable!”

I do not agree to “If several labs are able to … repeatedly show a significant effect, I don’t see anything wrong with that approach.” Applying significance thresholds makes cumulative knowledge unreliable. Even in ideal replication studies on true alternative hypotheses, significance is not to be expected, and nonsignificance cannot be used as a criterion to undermine the credibility of a preceding study. Replication should therefore not be judged based on the presence or absence of statistical significance. For details, I recommend reading the abstract or the content of Amrhein, Korner-Nievergelt & Roth (2017) and of Goodman, Fanelli & Ioannidis (2016).