How corruptive forces are chewing on science from the inside
On the scientific process and why only awareness and humility can yield great science
by Philipp Markolin & Laura de Vargas Roditi
The centrality of scientific papers
Scientific papers today are the primary way of distributing one’s research with the scientific community. Back in the 17th century, scientific papers were an invention that enabled researchers to inform the world of their results and were subsequently used to collect and disseminate research in a more systematic and organised manner.
Arguably, the very format of the scientific paper allowed science to evolve into a body of knowledge to build on and ever advance our quest for understanding the natural world.
Each paper is a small piece in a puzzle of unknown size or shape; the puzzle of reality.
And while we can’t see the whole picture, and maybe never will, pieces that do fit together grant us a better glimpse on that elusive reality underneath. This is what scientists mean when they talk about science as a body of knowledge and the best tool we have for approximating truth. Each little puzzle piece, once fit, allows us to declare: ‘Eureka! These belong together now.’ No matter what the ultimate shape of the whole puzzle really is, these parts will still be together.
Science is also hard work, contributing just one single puzzle piece (paper) can take years and be utterly frustrating because it might not fit anywhere at the time of publication. But maybe other people will read your paper and realize that it fits with the puzzle pieces they have found; and so a little picture emerges, then more puzzle pieces come in that fit, creating a paradigm (scientific consensus), and finally, decades down the road, a scientific theory can be formed (scientific theory = explains all puzzle pieces and makes accurate predictions for novel experiments). That’s how science moves forward, paper by paper.
Considering this, the centrality of the scientific paper in academic culture is no surprise. Papers are the currency and proxy which is used to evaluate scientists, award grants, create new scientific paradigms and form topical consensus to deliver as a public good.
If one were to taint these puzzle pieces, the scientific process would stagnate or even crumble. Unfortunately, we have reached a point where this fear is becoming reality.
Science under attack
In principle, we can stratify mechanisms which endanger the scientific process according to their originators; whether from within science itself or from outside of science.
Whole books (e.g Bad Science, The Demon-Hunted World, The War on Science) have already documented the plethora of atrocious attacks on science from the outside world; banal (yet incredibly harmful) misappropriations of scientific language by pseudo-science and snake-oil peddlers; unwise or calculated misrepresentations of scientific findings to justify policy; rightout malicious science denial of politicians, companies, activist or religious groups. I will refrain from listing abhorrent examples here for brevity and obviousness, and refer again to the excellent books above. The takeaway is:
There will never be a shortage of humans abusing or fighting science because of its inherent authority and claim to truth.
The sum of these outside forces endanger and take away public trust in what should be the most hopeful of our aspirations; doing science to make the world better. There is no method of thinking, no process of discovery, no self-correcting protocol that yields greater returns in knowledge, understanding and technological advancement than scientific inquiry. Science as a method has proven over and over again its capability to improve the human condition and tackle hard problems.
Yet little is being told about the corrosive forces inside science; many directly related to the centrality of this essential piece, the scientific paper.
As scientists, it is difficult for us to be anything but protective about the scientific process and the inner workings of science, especially since any acknowledgement of issues will be used by enemies of science to distort, discredit and sow distrust about the scientific enterprise as a whole. But as a wise man said, ‘it is far better to grasp the universe as it really is than to persist in delusion, however satisfying and reassuring’. So let us hold a candle towards the darker corners of today’s scientific enterprize.
The publishing game
Once a paper is ready, it has to go out in the world to take effect.
The way we distribute scientific information is in the hands of an oligopoly of big publishers, whose very profitable journal businesses coordinate peer-review (the practice of letting your competitors critically asses your scientific work), curation, editing and finally publication. They also charge scientists for submission of their papers, outsource peer-review to unpaid scientists, and demand hefty access fees from ‘customers’ who want to read the papers after publication.
In other words, scientists produce the product, pay to distribute it, do the quality control, and then have to pay for it again when they want access.
If this sounds absurd to you, you are not alone. But since we scientists need publications to advance science, apply for research grants or just to keep our jobs, we became dependent on the publishing industry with their own opaque rules and power dynamics.
There is plenty of evidence that the current publishing culture (described as “publish or perish”) has detrimental effects on the scientific process. For scientists, the pressure to publish papers in prestigious journals might force them to only pursue “sexy” stories, or cut corners to be first, or choose to discard experiments that do not fit the narrative (‘selective reporting’) . These biases might explain partly why some scientific fields are currently in reproducibility crisis. More on that later.
Yet individual researchers are not the only ones affected. For the public, journal paywalls limit free access to scientific information, which is a requirement for well-informed policy making. Topics like GMOs, quality of air & water, food labelling, treatment of chronic diseases, or alternative energy options are public concern-processes that must be openly discussed with public facts. Importantly, and often overlooked, the sheer cost of scientific or medical information behind paywalls is shown to asymmetrically hurt third world countries and minorities, thus contributing to racial, social and national inequality.
This is deeply troubling. In the age of the internet and social networks, where information is distributed freely, should publicly funded research really be controlled by the publishing industry?
In fact, the last 5 years saw an unprecedented engagement by scientists, institutions and importantly, members of the public, to reject access restrictions imposed by big publishers. This so-called ‘open access’ movement aims to make all research outputs to be distributed freely. Which is great, but the stakes are also high to do it the right way and not damage science in the process. And we’ve already messed up, big time.
For example, the unguided demand for open-access papers prompted the creation of a plethora of predatory journals of low quality with bad, fake or no peer review, which worsened the pay-for-play problem of getting papers published, as well as threatens scientific integrity as a whole.
Why did this happen? Well, established journals still serve as institutions of scientific credibility, they curate and quality control publications, an extremely important feature in a world of information abundance, fake news and filter bubbles, where search algorithms will find any result one wants to find, true or not. We cannot replace these institutions from one day to the next, nor should we want to. We should aim to fix what is broken.
Also, it would be far from me to solely blame the publishers for the state of science, when we also have to talk about the scientists themselves.
Gaming the system
“Measurement is not an alternative to judgement: measurement demands judgement: judgement about whether to measure, what to measure, how to evaluate the significance of what’s been measured, whether rewards and penalties will be attached to the results, and to whom to make the measurements available” — Jerry Z Muller, The Tyranny of Metrics
Science is expensive. When scientists experiment, there is usually a high chance that their hypotheses will be wrong, or things turn out unexpectedly. That is the nature of trying something new. Also, trying new things in itself can also be costly, because we need big expensive machines and technologies to try to do what nobody tried before, like colliding elementary particles in a supercollider, or probe the universe for gravitational waves, or develop therapeutic biomolecules hitherto unknown to nature.
Because the true value of any individual research project is diffuse, multi-faceted and hard to assess, making qualitative comparisons between various scientists is almost always difficult. Instead of facing this complexity, people responsible for resource allocation and funding for researchers primarily decide who to award resources by using a few simple numerical metrics;
- number of papers published (and the respective impact factor of the journal they are published in)
- number of citations or relative citations (h-index) of a researcher’s papers, both dependent on the number of publications
- amount of grant money/awards previously received by the researcher or his institution (also an iterative feedback of publishing a lot of papers)
In a profession built from and for the idealism of humble puzzlers, this output-centric shortcut might have made sense for science funding initially. Yet in a world where science is more and more transforming into a trillion-dollar humanity-spanning industry, these simple metrics cannot function any more. Also, scientists are by now keenly aware of what the current metrics are.
Rigorous science is hard work, it is all too human to want to take a short cut, especially if there is a big monetary incentive to do so. If publishing papers is the only metric that counts, people will optimize their work towards that, or worse, cheaters and frauds will gradually enter science to outcompete and replace “true” scientists so they personally can profit from this gameable environment.
Unfortunately, we observe both trends today, and they are chewing away science from the inside.
We have (an increasing amount of) scientists that have and will bend and tweak data, models or interpretations to fit narratives so their research can be published easily. Real scientists are supposed to be their own strongest critics, working through all eventualities to arrive at a standalone result; a finding they can defend to the best of their scientific ability and conscience. These paper tossers are also responsible for why big publishers have so much power upholding the current status quo. First they produce a flood of inflated papers of little real value, drowning out the competition and necessitating everybody else to play the publisher’s game of impact factors and journal prestige for their work to even be noticed. Second, they soak up funding and grants, which disproportionately falls to ‘hyperproducers’, and muddy the waters of what good research actually looks like for non-expert funding bodies. Third, they obstruct constructive change, since they made their careers on being the most prolific producers of papers (which is decidedly not the same as being the best scientists), it is their inherent interest to keep the ‘quantity over quality’ metrics in place.
Even worse than the tweakers, we also have the straightaway corrupt, cheaters who willfully engage in fraud to advance their careers, reputation, wealth, no matter the cost. While some estimate the percentage of these malignant elements in science to be small, my personal experience as a young academic is quite the opposite. In 2009, an anonymous survey found around 2% of scientists willingly admitting to fraud, and 15% alleged fraudulent practices about their colleagues. Since then, science has become even more cut-throat and we all are drowning in a poor-quality paper flood.
Both disastrous trends, the narrative tweakers and the cheaters, ultimately cause the successive replacement of the humble puzzlers of old by new stereotypes of career-scientists; the well-connected or politically gifted; the silver-tongued car dealers who repeatedly sell junk products; the fraud-savvy and immoral.
Worse, no matter if biased, fraudulent or ignorant, the papers these people (I henceforth refuse to call them scientists) produce serve as eternally flawed foundations no other can built upon, screwing up progress for real scientists as well.
Enter the reproducibility crisis
Reproducibility, or replication, of emperically gathered data is a foundational principle of the scientific method. In short, experimental results have to uphold given the same conditions independent of who performs the experiment. Because humans are flawed, full of subjective and cognitive biases, objective reality can only be gathered from independent validations, or better, people trying hard to flaw your results but failing to do so. Only results that can be experimentally disproven, but uphold continuous probing, are considered scientifically established.
It is no surprise that scientific reproducibility, an unambiguous hallmark of scientific quality, is worse in soft-science and fields of high competitiveness, of large money fluxes or of policy implications, because these fields attract the most fraudsters (biggest cake-size) and are the easiest to gamble (easier to bullshit in soft science). Reproducibility is still remarkably high in collaborative projects and hard sciences, like physics and chemistry.
Yet where science overall might be heading is foreshadowed by the incredibly wealthy and competitive field of cancer biology, the blackest sheep, where one reproducibility attempt showing only a laughable 11% of the most prominent/influential papers can be reproduced.
Given annual funding for cancer research is in the tens of billions, the reproducibility crisis in cancer research is in effect a giant waste of (mostly taxpayer or donation) money as well as a cynical scam played on a trusting society.
What can be done to fix science’s problems?
Solutions to the systematic problems of the global scientific enterprise are not easy to come by. Science functions in many aspects similarily to how a democratic society does, and might be as hard to fix once corrupted. We know there are no radical one-size-fits-all directives that can abolish a structural dilemma. Top-down approaches will not work. While tempting in thought, imposing any autocratic control mechanism over science would destroy the very essence of what is good about the scientific process. Instead, we argue that
It is up to individuals and institutions to improve areas of conflict and fight decay.
There are indeed a few hopeful trends we can discern:
- Calls for reproducibility are widely spread today by legacy journals like nature or science and have a big impact on public awareness. Thanks to these efforts, more institutions, journals, grants and awards committees are providing resources for reproducing already published science or incentivize reproducible research. That being said, reproducibility studies are still a rarety overall and the incentives provided are insufficient to cause a change in collective publishing behavior. Nevertheless, it is a necessity. A more hopeful role model in reproducible research might be the software community, where tools like github or project jupyter notebooks make reproducibility and re-usability of code easy and transparent.
- The open science movement, which aims at modernizing the way research is being communicated, makes it available for the public and also increases its transparency. It is also a major driver for open notebook science, citizen science, and aspects of open source software and crowdfunded research projects. It both adresses journal gatekeeping and reproducibility problems as well as creates opportunities to harness positive network effects through interactive collaborations. Hopeful open access examples include Nobel Prize winner Randy Schekman, who founded a peer-reviewed open-access journal called eLife, which became renowned almost immediately due to its backing by big scientific institutions that support its open access mission. Furthermore, we observe the ever increasing popularity and use of open access pre-print servers like arxiv or bioarxiv. These mostly grassroot movements are born out of scientific idealism but only thrive with collective awareness and adaptation by the scientific community, which luckily seems to be happening on many fronts.
- There are post-publication initatives like retractionwatch, pubpeer, bare-science or pluto.network which aim to clear out the cheaters once they passed through the peer-review cracks through community-evalutations or highlight value of under-appreciated studies to battle the flood of subpar papers. They speed up the self-correcting mechanism of science, provide new metrics to assess research quality and shift the incentives towards publishing quality instead of quantity. However, these novel initiatives are almost unknown to the wider scientific community as well as institutions, funders or the public. We hope that will chance soon.
- Lastly, effective and fair resource distribution for scientists and scientific projects is an open issue barely addressed. Even if we could eliminate cheaters who game the system, the way we currently fund scientists is inherently ineffective to maximize innovation or scientific output. Currently, the scientific community lacks ideas to solve the funding problem, while the overall societal framework is shifting away from having a multitude of universities with at least some resources to do cutting-edge research towards few well-funded ‘elite’ institutions and a lot of underfunded academics. Increasing scientific funding e.g by attributing more tax-payer money would not necessarily solve the problem, which is mostly rooted in a faulty winner-takes-all meritocratic distribution.
Scientific funding is largely distributed in a dominant meritocratic way, reflective of the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, smartness, efforts, willfulness, hard work or risk taking. […] It is very well known that intelligence (or, more in general, talent and personal qualities) exhibits a Gaussian distribution among the population, whereas the distribution of wealth — often considered a proxy of success — follows typically a power law (Pareto law). […] In particular, we show that, if it is true that some degree of talent is necessary to be successful in life, almost never the most talented people reach the highest peaks of success, being overtaken by mediocre but sensibly luckier individuals. — Pluchino A. et al, ArXiv, 2018
Indeed, when performing simulations accounting for the non-linear and messy path of scientific innovation, Pluchino et al. make a convincing argument that in order to maximize efficiency, funding has to be spread widely.
efficiency index Enorm are reported as function of the different funding strategies. The figure shows that
for increasing the success of a larger number of talented people […], it is much more efficient
to give a small amount of funds to many individuals instead of giving funds in other more selective ways.
Since rewards and resources are usually given to those that have already reached a high level of success, mistakenly considered as a measure of competence/talent, this result is even a more harmful disincentive, causing a lack of opportunities for the most talented ones. Our results highlight the risks of the paradigm that we call ”naive meritocracy”, which fails to give honors and rewards to the most competent people, because it underestimates the role of randomness among the determinants of success. — Pluchino A. et al, ArXiv, 2018
While enlightening, Pluchino et al.’s work is just a drop in the bucket of research needed to be done to understand and improve upon resource distribution on a individual, institutional and societal level, and this long overdue conversation has barely started yet. We hope it will soon.
Conclusion
Science is our most hopeful, multi-generational, civilisation-spanning quest for understanding of our world, our place in it and ultimately ourselves. It is also an inherently human endeavor, driven by individuals aiming, and often failing, to transcend personal flaws, biases and preferences in service of creating a body of reliable knowledge for all humanity.
We owe it to all of us to create an environment and incentives that bring out the best in scientists, or at least not stack the odds against them. Currently, external and internal forces make it increasingly impossible for scientists to become that necessary best version of themselves which can do great science. We all, scientist or not, share responsibility to fight systemic flaws in science however we can, or risk losing the only objective arbiter of facts and knowledge we have.
“I have a foreboding […] when awesome technological powers are in the hands of a very few, and no one representing the public interest can even grasp the issues; when the people have lost the ability to set their own agendas or knowledgeably question those in authority; when, clutching our crystals and nervously consulting our horoscopes, our critical faculties in decline, unable to distinguish between what feels good and what’s true, we slide, almost without noticing, back into superstition and darkness… -Carl Sagan
For us scientists, upholding our responsibility towards science might sometimes entail shining a light into the darker corners of what we want to protect.
Opinions expressed here are based on several years worth of academic experience and passionate discussions about the state of science with Dr. Laura de Vargas Roditi, making her an intellectual co-creator of this article. I also want to kindly thank Laura for sharing a treasure trove of resources and material for this article with me.
See also: Marc A Edwards and Siddharta Roy, Environ. Eng. Sci. 2017, Pluchino A. et al, ArXiv, 2018
This story is part of advances in biological sciences, a science communication platform that aims to explain ground-breaking science in the field of biology, medicine, biotechnology, neuroscience and genetics to literally everyone. Scientific understanding has too many barriers, let’s break them down!
