The Reproducibility Crisis of Medicine


The layperson is confused by contradictory and ever-changing reports in the media on the benefits of one thing or the other, so that we can’t even know what is good and what is bad. But most will be surprised that the situation in the scholarly journals is worse and experts believe there is a reproducibility crisis in science. And given the nature of the field, the reproducibilty problem in medicine is worse than in other areas.

A survey published in Nature in 2016 reported that researchers had been unable to reproduce over 70% of the findings of other scientists. Just a few years prior, researchers at the biotechnology firm Amgen reported only 11% of the pre-clinical cancer studies could be replicated.

The amount of flawed research seems too high. Certainly, there are unscrupulous scientists but the general view is that fraud is probably less than 2 to 5 percent, and the biggest reason behind irreproducible research is bad design and inherent biases. In a well-known paper, John Ioannidis, who is professor of medicine at Stanford University, claimed that most published results are false, and they merely present the prevailing biases in a dressed up manner. He further added that most clinical research is not even useful.

Ioannidis believes that the current medical research mainly takes care of “the needs of physicians, investigators, or sponsors” and for it to be useful again it will have to be “patient centered.”

To deal with inherent biases, of which some researchers may not even be consciously aware, there is need for new kind of design that approaches the hypothesis from different perspectives. There is also need for a clear identification of the underlying uncertainty model associated with the scientific study.

Mind-Body Connection

Allopathic medicine is based on treatment by the use of pharmacologically active agents or interventions to treat or suppress symptoms or pathophysiologic processes of diseases or conditions. It normally looks only at the body and uses protocols that are determined for large populations. There is little attention paid to how the mental state of the patient could affect the treatment of the disease. This is relevant as brain states and the immune system are linked.

According to a recent report: “Until recently, the central nervous system (CNS) was thought to be cordoned off from the peripheral immune system, reliant only on its resident immune cells called microglia. Peripheral immune-cell breaches anywhere in the CNS were considered signs of disease. But researchers now know that diverse immune cells — possibly by the millions — circulate in the cerebral spinal fluid (CSF) and live in the brain’s outer membranes even in healthy individuals.”

But these factors are not accounted for in studies on efficacy of drugs and, therefore, the test and control samples may not be robust enough to draw thepublished inferences.

The Problem of Logic

But there is a deeper problem to which almost no-one has given thought. This is that binary logic, which is at the basis of experiment design, is not optimal. Ternary logic is (and here is recent paper by me on this), and it requires the hypothesis to test between three classes that may be called “true”, “false” and “maybe”.

I am not speaking of multi-class classification which is common in data analysis; rather, I am speaking of an underlying triple logic associated with the fundamentals of the problem, which, to the best of my knowledge, has not been examined before. It is significant that a ternary classification of patients is used in at least one traditional medicine system (Ayurveda), but that specific use may not constitute TL in the sense we are using here.

Obviously, the use of TL will present many challenges of design. We will have to examine the analytical implications of modifying the current regime of two testing classes to three. The third class, if properly identified, may also make it easy to triangulate the study so that biases are minimized.

The use of ternary logic in experiment design may even help advance the understanding of the placebo and the nocebo effects in which there are therapeutic or unpleasant side effects of harmless pills based on the expectations of the patient.

The reproducibilty crisis of medicine, and the side-effects of drugs, is behind patients searching for alternative medicine systems like Ayurveda for the treatment of chronic illnesses.