How to Spot a Bullshit Study on the Internet

Every day I see links on Facebook claiming a new cure for cancer or some other absurd discovery. In the past, charlatans went town to town, selling fake cures to scam people for their money. Now, these swindlers exist online to garner clicks from people who don’t know any better. “Content” producers may not knowingly lie, but they greatly exaggerate science to make sensationalist headlines. To a well-trained eye, these inaccurate articles are easy to spot. However, not everyone is a nerd, so here are some tips to avoid falling for the clickbait.

Tips for Spotting a Suspect Study

1. It’s not even a published study

If the information in a study hasn’t been published, that’s because it’s bullshit. Scientists publish their research to peer-reviewed journals for a reason. Some “articles” have the audacity to tout a new discovery when it really isn’t even a discovery yet. A curious discovery may evolve into solid research, but hold your horses until it has at least passed through peer review.

2. The title of the article contains one of these phrases:

1. “…says science”
2. “According to science…”
3. “Scientists have been keeping X a secret for…”
4. “Crack the X code”
5. “The truth about...”
6. “Teens invent…”
7. “[any curse word]”
8. “You won’t believe…”
9. “You are being lied to…”
10. “Doctors hate…”
11. “We could be totally wrong about…”

These words don’t always mean that a link will be bullshit, but they are usually pretty strong indicators. Real information doesn’t need to hide behind sensationalist, clickbait headlines.

3. Small sample sizes

One good indicator of a potentially bogus study is a small sample size. If there is a claim that HIV has been cured in one patient with a breakthrough new treatment, chances are that this is insignificant. It’s easy for 5 things to be bullshit, but it’s really hard for 5,000 things to be bullshit. Even if an assay is 99% accurate, it will turn up false positives. It is better to look for repeated successes over large sample sizes.

4. Mistaking basic science for clinical science

The most common form of lying is twisting the truth. Sites tend to over-exaggerate the impact of a new discovery. Maybe a new drug is able to kill a very specific type of cancer cell in an in vitro study. You can bet that someone will assume cancer will be cured within the month. However, there is a huge gap between a basic discovery and a functional drug. Only about 1 every 5,000 potential drugs wind up making it to market, and only 1 in 1,000 even make it to human trials. While basic science is hugely important, it may take these discoveries nearly 20 years and over a billion dollars to reach a pharmacy.

5. Researcher bias

Every human has bias. Good scientists will try very hard to mitigate their own biases. But some research can come with clear conflicts of interest. An oil company claiming its pipelines don’t devalue property may not be the most reliable study. Place higher value on third party research.

6. Correlation vs. causation

This is one of the greatest sources of error in many poor articles. Biology is difficult to study because there are millions of processes going on in the human body at once. Some of these are bound to correlate, even if they are unrelated to each other. Always look for a mechanism of action. If one event does indeed cause another, we will have some understanding of how exactly they influence each other.

7. Misinterpretation and Speculation

Just because we have found a way to regulate one gene which helps regulate the aging process does not mean that we have reversed aging. Biological processes are complex, and it is a logical fallacy to jump to conclusions from one piece of evidence.

8. Poor experimental design

A lack of a control group, failing to use blind testing, and cherry-picking results are all examples of a bad experimental design. Data is worthless if there is no control group. Blind testing is necessary for reducing patient bias. Cherry-picking only desirable results or desirable subjects is just a way to lie with statistics.

Lastly, a surprising number of studies may be wrong

The scientific community has been scrambling to rethink some of its best practices, with new studies suggesting that the majority of published research may be impossible to reproduce or false. In 2005, John Ioannidis published a shocking paper called “Why Most Published Research Findings are False.” In a review of reputable journals, scientists were unable to obtain the same results as the original researchers when trying to replicate studies. This was the case for over half of all published studies. While the scientific method has self-correcting mechanisms, these studies and others have caused researchers to pause and rethink their standards. For more about the flaws of the scientific process, see this article by William Wilson. The way we conduct and report science is not perfect, and we need to constantly strive to obtain more accurate results for the good of our species.

“In the long run, good science drives out bad.” — Martin Gardner

For more posts about disease and biology, subscribe to Cell Your Soul. Feel free to comment below or message your feedback!