3 Reasons to Not Blindly Trust Peer Reviewed Research
An opinion essay discussing the imperfect process of publishing research.
Peer review is a critical part of the scientific process, serving as a quality control mechanism to ensure that research is rigorous, well-conducted, and scientifically sound. It is a process introduced to hold researchers accountable as a check for honestly and accuracy.
Every time a researcher submits a paper describing any research conducted to a peer reviewed conference or journal, it enters a screening process. Many academics who are respected in their fields volunteer for these reviewer positions, and are usually qualified to at least assess if the techniques applied in the project were done correctly to provide valid results. If the paper meets these reviewers’ standards, it then goes into a revision process that typically takes months to address any flaws or alterations the reviewers believe are necessary to complete the work and communicate it well to the public. Once a paper has been iterated enough times, the publisher accepts the quality is at a standard of their usual publications. Then, the paper is broadcast to the world.
However, simply because a study has undergone peer review does not necessarily mean that its findings should be blindly trusted or considered absolute truth. There are numerous examples of peer-reviewed research that have later been found to be flawed or even fraudulent, that bring valid concerns about the reliability of the scientific process. The good news is many of these papers do not last in credibility, are quickly outdated by higher quality research, or are redacted by the author(s) or publisher after the mistake is identified. The bad news is the findings may be spread quickly by journalists and reporters that have large audiences before the mistake is found.
So what are some examples? I composed a short list of papers that have been published after peer review, but have been identified as making critical mistakes:
- Article researching stem cells in the heart retracted for curated lab results
- Article researching plagiarism that was retracted because of plagiarism
- Article claiming grizzly bears become diabetic during hibernation retracted due to falsified data
These are just several of many papers retracted every year due to identified misconduct or false results. Some research can still be misleading even if there lacks major issues to retract it. For example, a paper on eye movement desensitization and reprocessing (EMDR) treatment for depression in heart attack patients is still considered valid research, but has imperfect statistical analysis. They make an assumption about (or fail to address) whether there is equal variance in their treatment and control groups when performing a two sample t-test (two sample t-tests are performed differently depending on equality of the population variances). Additionally, they make causal claims in their conclusions on observational data without addressing potential biases. Even so, this paper still has plenty of good results to contribute to the scientific community. I do not believe it should be redacted, but I present this paper and the papers listed above to show that research, even when considered valid, can have flaws.
So how do these papers get published if there major mistakes?
- Bias: Peer reviewers are typically chosen from the same scientific community as the authors, and may have personal or professional relationships with them. This can lead to a conflict of interest and a tendency to be more lenient in the review process. Reviewers have the ability to abstain from review of a paper due to a conflict of interest, but in less obvious cases, it may not be required.
- Bad journals: Varying reputation in journals is a known fact in the scientific community. The main metric to measure a journal’s prestige is its impact factor. Although, a journal having a lower impact factor does not imply it is unreliable. For the most part, researchers are connected with their peers in the field and are aware of what journals are trustworthy, although, reporters that communicate results from research to the public may not be aware of this. Regardless, there are still predatory journals that are not transparent about their process, put in “copy-proofs” that prevent plagiarism checks, or exist just to take money from researchers. There are plenty of ways for a journal to have unethical motives, and organizations are trying to combat this.
- Mistakes: Reviewers are often busy with their own research and may not have enough time to thoroughly evaluate the work they are reviewing, leading to missed errors or oversights. While it is expected that reviewers take their position seriously, we are all human and are bound to make mistakes. Also, deliberately false data or results introduced into a study can be hard to detect from a third party even if they are an expert in that field.
While peer review is an important part of the scientific process, it is not a guarantee of the reliability or validity of research. There are many factors, including subjectivity, biases, and the limitations of the review process, that can lead to peer-reviewed research being flawed or even fraudulent. As a result, it is important to be critical and cautious when interpreting research, and to consider a range of sources and perspectives before accepting any claim as true. This can include gaining knowledge in a field for better interpretations, finding several papers in agreement from similar experiments, and not taking statistical interpretations at face value, especially when explained by someone other than the authors.
In short, peer review should not be the sole basis for determining the reliability of research, but just one of several quality control measures that should be used to ensure the reliability of scientific findings. Peer-reviewed research brings a strong filter to encourage exceptional research, but should never be viewed as absolute truth.
Joshua Anderson is currently working towards a Ph.D. in Intelligent Systems at the University of Pittsburgh researching fairness in medical AI models.
Liked what you read?
Subscribe to my news letter here and
Check out my other articles: https://medium.com/@talkai
Disclosure: some of this article was written with the help of AI-assistive technology
