Reproducibility Crisis: The State of the Science
In this extract from Defense of the Scientific Hypothesis, Bradley Alger explores the phenomenon that continues to plague scientific research and asks, “how big is the problem, really?”
The news is everywhere: biomedical science is in trouble. Scientific investigators cannot confirm (i.e., reproduce) published findings. Reports of irreproducibility are proliferating, and the trust that scientists and the public have in scientific knowledge about the world is eroding.
Smoldering concerns about repeatability in science burst into flame after scientists at Bayer HealthCare Pharmaceuticals and Amgen vented their frustrations at going down expensive and time- consuming blind alleys in their search for new cancer therapies. The company scientists reported that they could not repeat the findings of 65– 90% of the preclinical science studies that they evaluated. Normally, commercial laboratories extend and amplify leads provided by preclinical research, develop and test candidate drugs, and eventually bring new medicines to market, so the apparent unreliability of the evidence set off alarms throughout science and attracted the attention of the popular press. Columns in The Economist (“Unreliable research: trouble at the lab”), Los Angeles Times (“Science has lost its way, at a big cost to humanity,”), New York Times (“Why do so many studies fail to replicate?”), and many others made the public aware that all was not well in the laboratory.
The frantic tone of the newspaper headlines implies that reproducibility is the highest goal of science — that the main purpose of scientific inquiry is to generate reproducible results and that the more reproducible the result, the better the science. This isn’t true. As science tries to learn about the world, its objectives include acquiring and interpreting data, explaining its findings, and making predictions of how future experiments will turn out. Reproducibility is a scientific virtue, a core principle that helps science achieve its goals; it is a highly desirable property of research; it is not the main thing.
…
While there is universal agreement that scientists cannot always reproduce published reports, there is disagreement about the causes of irreproducibility and how common it is. How big is the perceived problem?
Reproducibility is a scientific virtue, a core principle that helps science achieve its goals; it is a highly desirable property of research; it is not the main thing.
In May 2016, Nature conducted a survey of its readers that drew 1,576 responses. When asked, “Is there a Reproducibility Crisis?” 90% of the respondents said that there was either a “significant” (52%) or “slight” (38%) crisis, and only 7% felt that there was “no crisis at all.” Nature is aimed at a scientifically trained audience…the results probably convey a fair sense of the community attitude: namely, that many scientists feel that a significant fraction of the published work in their field is not reproducible.
Then there are those who question the concept of a Reproducibility Crisis, pointing out that it can be extremely difficult to duplicate the exact conditions of a published study. For instance, in fields such as psychology, the precise social context constitutes an amalgam of potentially crucial features — interactions among the individual investigators, the particular environmental setting, etc. — that are probably impossible to duplicate.
In any case, it is scientists, not non-scientists, who discover and rectify the errors that scientists make. This is the self-correcting property of science; finding and fixing mistakes are signs that all is well. Last, complaints about reproducibility are nothing new; they have been around for hundreds of years, and yet nobody denies that we have amassed a huge and rapidly expanding trove of genuine, reliable knowledge about the world. Many commentators think there is nothing to worry about.
So there are two sides to the debate, and we’re left with the question, “Is there is a crisis or not?”
…
When it comes to science, reproducibility is not everything, and losing sight of that fact may have unforeseen consequences. In 2015, the US Senate Committee on Environment and Public Works (EPW) passed the Secret Science Reform Act of 2015, which prohibits the Environmental Protection Agency (EPA) from “proposing, finalizing or disseminating” a regulation “unless all scientific and technical information relied on…is the best available science, specifically identified, and publicly available in a manner that is sufficient for independent analysis and substantial reproduction of research results.”
It is scientists, not non-scientists, who discover and rectify the errors that scientists make. This is the self-correcting property of science; finding and fixing mistakes are signs that all is well. Complaints about reproducibility are nothing new; they have been around for hundreds of years, and yet nobody denies that we have amassed a huge and rapidly expanding trove of genuine, reliable knowledge about the world.
Given the burgeoning public anxiety about a crisis of reproducibility, you might think this congressional call for transparency and reproducibility is commendable. If so, you’ll be surprised to learn that several senators and others concerned about climate change, and clean air and water, are exceedingly worried by the Act and are opposed to it. In holding up reproducibility as a defining characteristic of the “best available science,” quantities of valuable, though unavoidably irreproducible, scientific data could be forbidden to the EPA. For instance, the British Petroleum Deepwater Horizon oil spill in the Gulf of Mexico in 2010 was an environmental disaster. It was the largest such spill in history, the subject of intense scientific scrutiny, and the source of enormous amounts of valuable data. It was also a “one- time event” and is, therefore, irreproducible by definition. The Act also calls for transparency, conceivably implying that all of the data used by the EPA, must be publicly available for independent reanalysis. Making everything transparent would be extremely onerous and expensive. Finally, satisfying the transparency requirement could be impossible if confidential patient records were involved.
In commenting on the Act, American Association for the Advancement of Science (AAAS) President Geraldine Richmond noted that “while transparency and reproducibility are of utmost importance to the scientific community, this mandate…is overly broad and will have severe unintended consequences.” It may be too optimistic to expect the US Congress to draw and respect the fine distinction between “utmost importance” and “overly broad” when the political lines are sharp and the stakes are high. If Congress and the public had a better understanding of science, including a realistic appreciation of the roles and limitations of reproducibility, such difficulties might be headed off.
…
Reproducibility is obviously a multidimensional goal and will not be achieved by any one set of strategies. There is a wealth of expert opinion on rectifying its qualitative and materials-related deficiencies. Far less attention has been paid to possible weaknesses in scientific reasoning. Good experimental design requires clear scientific thinking and skill in using the Scientific Method and explicit hypothesis-based reasoning. Reproducibility is unquestionably an important challenge for science, but how much is too much? How much genuine irreproducibility should we tolerate? “Zero” is obviously unattainable, but what is a reasonable amount? Without good estimates of these factors, the extreme anxiety that has been expressed about a crisis seems at least premature.
Bradley E. Alger (AB, UC Berkeley; PhD, Harvard), conducted some of the initial in vitro studies on synaptic plasticity in the brain for his thesis. In 1981, after serving as Roger Nicoll’s first postdoctoral fellow at UC San Francisco, Alger was appointed Assistant Professor of Physiology at U Maryland School of Medicine. He did research and taught until becoming Professor Emeritus in 2014. Alger’s laboratory published over 100 research articles and made fundamental discoveries regarding “the brain’s own marijuana.” While lecturing on scientific reasoning, he saw the need for a book like this one.
