Science Is in a Worsening Crisis
The crisis concerns reproducibility and incentives in the enterprise of science
Peer-reviewed scientific research published in respected scientific journals is the standard by which the enterprise of Science moves forward. Science may range from more fundamental fields such as in High Energy Physics to understand the basic constituents of matter simply understand the universe to more practical fields such as Biomedical Sciences that drive the formulation of policy and government regulations that are enforced by the Food and Drug Administration.
Even less applied and fundamental breakthroughs in physics can lead to revolutions that change the world and affect day-to-day living. Witness how Solid State physics lead to the invention of the transistor, which in turn transformed the world through the Internet.
However, despite all the progress enabled by the enterprise of Science, we are facing a crisis and it’s worsening. I will consider here the main constituents of the crisis:
- Incentives for those doing Science
What is Science?
Let’s briefly consider what Science actually is and aspires to be by defining specific criteria, before considering exactly why we’re facing a crisis in Science.
The philosopher of science Karl Popper formulated a very succinct and powerful definition of science:
- Science is what is falsifiable
Embedded in this simple statement are many important issues including the following:
- We must be able to measure
- We must be able to reproduce
- We must be able to falsify
With each of these, there are a host of difficult problems.
- Measurement may be difficult or incomplete since the real world is messy and so our instruments and recording devices are imperfect.
- Reproducibility may be difficult because experiments can be extremely expensive and precise details of methods used may not be disclosed.
- Falsifiability may be difficult because it may involve the proper application and interpretation of statistical concepts such as confidence intervals and p-values.
Philosophy of Science
In addition to the problems here are important issues that may start to delve more into the Philosophy of Science. For example, consider the following.
- Scientific truth is only arrived at asymptotically, but there is never “verification” of the truth since a single example can falsify a model.
- A single Black Swan can invalidate the belief or scientific model that “all swans are white.”
- Scientific truth is only relative to a specific context.
Assuming we can even formulate scientific theories based on empirical data with statistical significance, truths within those theories could be invalidated by changing the domain of application.
For example, assuming we have truths in Classical Physics and Newtonian Mechanics, they may be invalidated:
- at high speeds since we must account for Special Relativity
- at very large scales since we must account for General Relativity
- at very small scales since we must account for Quantum Mechanics
Measurement may be “merely” a technical or engineering concern even though it can be quite complex and difficult such as the calorimeters and drift chambers in the detectors at the Large Hadron Collider at CERN. It’s possible to expound at length in the Philosophy of Science concerning falsifiability with respect to probabilistic or statistical concepts and dive down the rabbit hole with respect to broader issues in the nature of scientific truth.
However, one thing that is relatively easy to define, simple to understand, and easy to control for is reproducibility. Unfortunately, the problem with peer-reviewed published scientific research is that it cannot even be reproduced.
The Crisis of Reproducibility
Survey on Reproducibility
The following article was written by Monya Baker and published in Nature concerning results from a survey of 1,576 researchers who took an online questionnaire concerning reproducibility.
- 1,500 Scientists Lift the Lid on Reproducibility
Nature. July 28, 2016
It’s a very eye-opening article with many nuanced results and interpretations, but the main takeaways are:
- 52% believe there is a significant reproducibility crisis
- 38% believe there is a slight crisis
Since the terms “significant” and “slight” may mean different things to different individual scientists and also their specific field of research, it’s hard or not useful to get more precise than this. However, it is astonishing since:
- A vast majority (~90%) of researchers may agree there is a reproducibility crisis.
When asked about “why” there’s a reproducibility crisis, greater than 60% of the respondents attributed it to two specific factors.
- Pressure to publish
- Selective reporting
This is astonishing and points to academic incentives for doing good science, and what it means for the state of Science.
Pressure to Publish
Although the phrase used is “pressure to publish”, in effect it is “Publish or Perish.”
This is more acutely felt by more junior scientists, starting from graduate students at the very bottom ranging to post-docs and junior faculty. They must produce something since those publications are proof that they’ve been productive. Absent that proof, they cannot sufficiently validate themselves and thus both their financial status and careers may be jeopardized, not to mention the personal toll such as self-esteem. This is strong pressure indeed.
The problem with publishing is that it assumes a “positive result”, in that you must demonstrate you found something. If you failed to find something or can definitively show that there is nothing to find, it is called a “negative result” in contrast. Negative results tend not to be publishable, even though they may be just as important to scientific progress.
An example of negative results is the constant streaming of “negative” results from large scientific projects such as the Search for Extraterrestrial Intelligence (SETI). The very fact there is nothing to be found points to the lack of advanced intelligent life in the universal, and this is extremely important.
There have been some attempts to create journals of “negative” results and there even exists one of exactly that name, Journal of Negative Results published by Springer. Unfortunately, it has ceased publication as of September 1, 2017.
It can be construed that selective reporting is intimately related to not publishing negative results. There is relatively little incentive for referees or the larger scientific community to ask for negative results and investigate them deeply compared to tangible positive results.
Although it is true that the researcher can make statements about the lack of findings, it’s usually only done so secondarily to enhance the context of reporting or discussing other more significant findings. A comprehensive listing of a lack of findings is simply not publishable, nor even useful.
When polled about reproducibility, over 60% of the respondents said both funders and publishers should do more to address reproducibility. The top five ways to do so on the researcher level may include:
- Better understanding of statistics
- Better mentoring and supervision
- More robust design
- Better teaching
- More within-lab validation
Problems in Scientific Research Beyond Reproducibility
What Other Problems Exist?
Reproducibility is a core problem in experimental science and two main causes were cited, the pressure to publish and selective reporting, as well as some methods to address them.
However, let’s assume we can solve the reproducibility of experimental data well or that we focus on scientific research that may not be so critically dependent on empirical data such as mathematics or theoretical physics. There still exists a host of problems with the enterprise of science.
Science in the Age of Selfies
An illuminating opinion piece is linked below, written by applied mathematics professors Donald Geman of Johns Hopkins University and Stuart Geman of Brown University, both members of the National Academies of Sciences in the Applied Mathematical Sciences division.
- Opinion: Science in the Age of Selfies
Proceedings of the National Academies of Sciences. August 23, 2016.
Their main contention is how the overall pace of fundamental progress in Science has actually slowed down because people are more worried about taking “professional selfies” as superficial and incremental status reports while they’re too wired to the greater scientific community, and how this may hinder deep thinking.
This is so despite there being more scientists alive now than ever, despite more research dollars being spent, and despite how much futurists and transhumanists may hype the future impending and inevitable “Singularity.” It’s important to understand that we do not passively wait for the future to just happen to us, but rather we create it.
Their views concern three broad reasons why Science may be slowing
- Excessive Collaboration
- Day-to-Day Realities
- Quantity Over Quality
The world is now more connected than ever and the ease of travel along with cheap high-quality communications tools such as video, email, and even social media permits instant communication that is actually leading to a problem of “excessive” collaboration.
Obviously collaboration is necessary especially in large scientific projects like the Human Genome Project, but too much collaboration may stifle or slow scientific progress if it takes away from time and brain cycles that otherwise could’ve been diverted to deep thinking at the individual level.
The daily grind mentioned is not so different from the discussion about the underlying causes of the reproducibility crisis. Researchers may be forced to spend too much of their precious time on the following.
- Submitting a high quantity of papers to show progress
- Writing research proposals to secure funding
Original thought that is far from the mainstream may be required for making progress on complex, deep, or seemingly intractable problems, but too much originality may involve excessive risk precisely because it is far from the mainstream and thus cannot produce a stream of incremental published research or lead to government funding.
The consequence is that creative scientists who otherwise would pursue impactful work and make significant progress to forge ahead in yet-unknown scientific fields are unfortunately instead relegated to defined and narrowly-focused roles effectively as government contractors.
Quantity Over Quality
Finally, the incentive structure has changed to produce superficial and incremental results, usually driven by “Big Data” and Machine Learning rather than hypothesis-driven science.
Although Machine Learning has led to some incredible performance in narrow domains, it’s ultimately detecting patterns or fitting curves in high-dimensional space and is not of the same qualitative nature as hypothesis-driven science, is actually what’s required for consequential innovation and transformative ideas.
Awareness of The Crisis Is Crucial
In summary, the reasons why there’s a crisis in science and scientific research are as follows.
- Problems with Reproducibility
Concerns narrower problems of reproducibility related to empirical results and specific methodology.
- Incentives in the Enterprise of Science
Concerns broader problems of the system of incentives and slowing of scientific progress in general.
Simply being aware of these problems, defining them tangibly, and communicating them is an important first step towards solving them, even if we do not or cannot solve them outright.