Medically Clear #19: On Truth in Research
Perhaps you read in the New York Times about the slew of promising anti-aging drugs in development.
Listen to the podcast here or in iTunes
Metformin (Glucophage) is one example. Metformin has been used since the 1920s to help control diabetics’ blood sugar levels but it may actually do much more — perhaps prolonging life in diabetics and non-diabetics alike. Purportedly, metformin modulates metabolism and cellular damage through its effects on the liver controlling glucose (sugar) production. Biologically plausible? Yes. But do I advise adding metformin to your regiment of medications and supplements? Not so fast.
It’s an unfortunate reality that many biomedical research “findings” are ultimately proven to be illusory. This goes for drugs (remember when Vioxx was considered a safe treatment for arthritis?), dietary associations (oat bran to prevent heart disease?) and diagnostics (when exactly should you start getting mammograms?) How frequently do facts become fiction? Well, sadly, quite frequently — in fact, “findings” in medical research may be only slightly more reliable than Donald Trump’s “facts” and only about as good as sport pundit “expert” picks against the spread. Feeling doubtful Thomas? Here’s a look at the evidence behind medical evidence.
First, let’s define what we mean by a “research finding.” A research finding is any relationship discovered in a study that meets the criteria for statistical significance. The typical standard for this is a “p value” of less than 0.05, which essentially means that there is only a one in twenty chance that the research finding was due to chance rather than a real association. But while 95% confidence in the truth of a finding may seem intuitively appealing, there’s evidence that it may not be conservative enough.
Consider a 2005 paper by John P.A. Ioannidis entitled “Why most published research findings are false.” Within, Ioannidis builds a theoretic model of the chances of obtaining a true research finding starting with sample size (how many patients are in the study) and p-value. He then mixes in additional terms to account for possible confounders that could cause a finding to be falsely true. These include investigator bias (recognized or not), the probability, prior to study, of the association being true and the number of different possible associations being tested. From his model, Ioannidis concludes that the chances of a finding being a true finding range from about 85% to lower than 20%! Eighty-five percent if the study was a well-controlled randomized trial of an association with reasonably high pre-study odds of being true; 20% or less if the study was retrospective (backwards looking) and examined rather broad associations such as diet and disease (think processed meat and cancer risk) or thousands of associations at once (think genetic-discovery oriented research where 30,000 genes may be tested to find 30 or so true culprits.)
Subsequent theoretic analyses have lent support to Ioannidis’ notion and generalize that no better than 50% of research findings are actually correct.
Indeed, in our recent history, there is a litany of research findings that could not be replicated or were outright refuted (e.g., The Lancet’s vaccines-cause-autism study). As such, the topic of how to distinguish the truth from the false positive has picked up considerable steam.
If this has you feeling skeptical about the health recommendations you receive and the value of biomedical research, you are not alone. Fortunately, this is not a presidential debate situation where all conclusions should be considered false until impartially verified. There are principles that can help both researchers and the public arrive at a better state of biomedical knowledge. And here’s where you come in: Do not simply assume that all research follows the following principles, be an informed consumer and learn to assess the quality of the “findings” you hear in the news.
So, here are those principles:
1) Look for numbers; bigger is better. The work of Ioannidis and others has demonstrated that small numbers of observations or study subjects are more likely to have spurious results and be non-replicable. Place more credence in studies that have thousands, or tens of thousands of subjects rather than those with just a handful.
2) Be wary of bias. Any studies published by entities with possible conflicts of interest or on a topic of newsy interest have a higher chance of false findings. This is not to say that every drug company study is rigged, but caution is advised in interpretation.
3) Perform a ”does it makes sense?” test. The probability of a particular truth in medicine before a study is done greatly affects the likelihood of a finding being actually true. This principle is known as face validity. For example, just because some people who eat a lot of ice cream are thin does not mean that ice cream is an effective weight loss tool.
4) Ask if the findings are randomized and replicated? In 2015, there were a series of studies, all randomized trials that all demonstrated the benefit of a new clot retrieval technique for certain types strokes. These studies all employed the best type of study design (randomized controlled trial) and had strongly positive results that were repeated in multiple other separate investigations. We can believe these results.
5) Five Sigma. Different scientific disciplines use different thresholds for defining a research finding. In medicine, the threshold is two sigma, which is two standard deviations or 95% confidence. In physics, however, it is five sigma, which equates to 99.9999997% confidence. If you see a finding in medicine that meets this five sigma criteria, you better believe that Sir Isaac Newton would approve.
So, before you start popping metformin, or whatever trendy supplement or super food, it is worth considering the spotty record of science’s record in discovering truth. Fortunately, you can also take heart in the fact that science remains the most vigorous supervisor of its own truths — false findings are discovered and discarded, and the total body of evidence moves forwards. It may be messy, but ultimately, it gets it right the vast majority of the time.
