Fact checking a VC’s tweets about COVID-19 clinical trials
Keith Rabois versus Zach Weinberg. Do Rabois’s claims stack up?
A twitter argument inspired the following summary of COVID-19 clinical trials as well as my effort to fact check statements made by VC Keith Rabois in his exchange with healthcare CEO Zach Weinberg.
Here are the tweets captured by the account @vcstarterkit:
I’ve paraphrased the battle by removing insults and typos in order to better clarify the arguments presented.
Battle Recap:
Keith Rabois tweeted that “randomized controls are horrible ideas,” and stated that Fortunately, France, NY and UCSF are not listening to the nonsense about randomized clinical trials. After this, Rabois concluded that the *current* FDA allows field trials to substitute for a randomized double blind study.
Zach Weinberg hotly questioned these statements and posited that, “this stuff” (meaning statements by Rabois) is dangerous and wrong and healthcare is different than tech.
The statements Keith Rabois made are inconsistent with what I’ve experienced as a biomedical scientist and public policy advocate.
This argument prompted me to ask the question: have clinical trial best practices changed during this healthcare crisis or are Rabios’s statements unfounded?
A little about me: I’ve worked as a scientist for 10 years, including as a researcher at St. Jude Children’s Research Hospital and at a biotech startup. I’ve also advocated with Life Science Tennessee in Nashville, TN, and in Washington D.C. in favor of 21st Century Cures Act (a 2016 bill that changed FDA regulations to encourage faster clinical trials in special cases). I am familiar with how clinical trials are regulated, developed, and run. Given my background, I want the facts!
For my own clarity on this topic, I set out to do three things:
- find and organize COVID-19 clinical trial information
- understand if the clinical trials in France and the US are randomized
- determine if changes were made to the best practice clinical trial process for COVID-19 clinical trials
In the sections below, I’ll explain my methodology, review source material, then compare my findings to the claims made by Rabois on Twitter.
1. Organizing COVID-19 clinical trial information
My first step was to organize what is known about the COVID-19 clinical trials that are underway in France and the US (the places Rabois mentioned) by reviewing multiple primary sources (when available).
Here is a summary of what I found regarding COVID-19 related clinical trials in France and the US:
There are two connected clinical trials to treat COVID-19: SOLIDARITY and Discovery
SOLIDARITY: Conceptualized by the World Health Organization (WHO) to be executed by many countries.
How it works — patients are enrolled in a randomized study and are treated with one of four therapies:
- remdesivir — an experimental antiviral compound
- chloroquine and hydroxychloroquine — malaria medications
- a combination of lopinavir and ritonavir — two HIV drugs
- lopinavir and ritonavir plus interferon-beta — an immune system messenger that can help cripple viruses
Discovery: A branch of SOLIDARITY that was conceptualized by the French health care firm Inserm and is being executed in France and seven other countries.
How it works — Patients are enrolled in a randomized study and treated with the same drugs as above (except chloroquine).
Therapies in the Discovery trial:
- remdesivir
- lopinavir and ritonavir
- lopinavir and ritonavir plus hydroxychloroquine
- lopinavir and ritonavir plus interferon-beta
Gilead clinical trials in California: Remdesivir
- A Phase III randomized study in patients with moderate and severe COVID-19
The National Institute of Health (NIH) and National Institute of Allergy and Infectious Diseases (NIAID) clinical trial: Remdesivir:
- A multi-institutional, Phase III, double-blind, randomized, placebo controlled clinical trial
- Enrolling patients at 40 locations, beginning at the University of Nebraska Medical Center and including The University of California San Francisco (UCSF)
Regeneron and Sanofi clinical trial in New York: Kevzara (SARILUMAB)
- A Phase II/III randomized, double-blind, placebo controlled trial using a monoclonal antibody targeting IL-6 in patients with severe COVID-19.
A convalescent plasma clinical trial with an unknown partnership out of New York:
- Injecting COVID-19 positive patients with plasma from patients who previously had COVID-19.
- Physicians can only administer this treatment to patients with life-threatening disease.
An investigational vaccine clinical trial in Seattle:
- Three groups of healthy individuals will be injected with a messenger RNA (mRNA) vaccine to compare different doses. The groups will receive either 25 mg, 100 mg or 250 mg of vaccine and safety and immunogenicity (immune response) will be evaluated.
- This Phase I trial will be administered to 45 patients, is non-randomized, and is an interventional clinical trial.
One clinical trial to treat pneumonia induced by COVID-19: Actemra® (tocilizumab)
- COVACTA: A clinical trial partnership between Genentech and Biomedical Advanced Research and Development Authority (BARDA) in New York
- Phase III randomized, double-blind, placebo controlled clinical trial
2. Are France and the US implementing randomized clinical trials?
Now that I’ve organized the clinical trial information for quick reference, I’ll break the statements from Keith Rabois into parts, and fact check them one at a time.
Recap: Keith Rabois tweeted, France, the NY and UCSF are not listening to the nonsense about randomized clinical trials and the *current* FDA allows field trials to substitute for a randomized double blind study — two tweets
Statement #1: “France, the NY and UCSF are not listening to the nonsense about randomized clinical trials.”
Facts check #1:
1. 🛑 The clinical trial in France is randomized and regulated.
2. 🛑 The plasma clinical trial in NY is FDA approved under a special provision “through the process of single patient emergency Investigational New Drug Applications (eINDs) for Individual patients under 21 CFR 312.310.”
3. 🛑 UCSF has not launched a clinical trial (according to publicly available information) but is participating in Gilead’s remdesivir clinical trial.
- Note: The other US clinical trials included clear statements that they are randomized clinical trials. One clinical trial (mRNA vaccine) was not randomized in Phase I, but they included internal study controls.
3. Have clinical trial practices changed due to COVID-19?
Statement #2: “the *current* FDA allows field trials to substitute for a randomized double blind study.” — Keith Rabois
This statement surprised me because it implied that clinical trial regulations are changing, so I broke it down and fact checked it.
“The *current* FDA”: I’m assuming Mr. Rabois meant the FDA during the COVID-19 crisis.
“allows field trials to substitute for a randomized double blind study”: Randomized, double blind clinical trial studies are the gold standard of medical research.
My interpretation of this statement is that Mr. Rabois is suggesting that the gold standard of clinical trials is being replaced by field studies during the COVID-19 crisis.
Next, I wanted to understand what a “field trial” is, since this is not a common term in a clinical trial context. I’m assuming that a field study is similar to what Mr. Rabois referred to.
Field study: studies carried out on the ground, or in the field. Community studies allow for an analysis of a large population of people exposed to a similar agent. (example: fluoride in water).
Facts Check #2:
🛑 I have not found any evidence that a field trial (or study) can substitute for a randomized double blind study, specifically in the case of COVID-19.
However, it is possible that Keith Rabois was pointing to the NY plasma study as an example of a special type of clinical trial. I’ve broken down its nuances in the next paragraph.
The NY study procedure is the least rigorous clinical trial protocol listed in this article of US clinical trials, but it has the most strict use case, meaning that clinicians are expected to apply this method only if the patient is dying, not as a preventative measure. Therefore, it would be difficult for clinicians to collect clinical information in a large number of patients quickly — an important aspect for statistical analysis of clinical trials. Enrollment in this clinical trial is so limited it would be difficult (nearly impossible) to enroll enough patients at once to complete a randomized, double-blind placebo controlled study. Therefore, this treatment is not part of a randomized, double-blind placebo controlled study, and is part of a special type of clinical trial.
In conclusion:
In my opinion, healthcare decision makers, like the WHO and FDA are applying logical, best clinical trial practices to the process of collecting information about COVID-19 treatments while also emphasizing the speed necessary to care for those infected.
There are no examples of the FDA allowing field trials to substitute for clinical trial data in COVID-19 related treatment processes.
While I don’t agree with Rabois’s approach to making strong public statements about COVID-19 clinical trials without citing his sources, there has been one silver lining from this twitter feud: the concerning claims of the argument pushed me to learn about COVID-19 clinical trials, to organize my thoughts and fact check these statements.
Additional resources:
The National Institute of Health (NIH) website with a running list of clinical trials:
A running list of groups working on COVID-19 treatments with links to primary sources:
The US and France are not the only countries establishing clinical trials for COVID-19. Additional COVID-19 clinical trial information from the UK, Canada and Europe can be found here:
Note: all of these opinions are my own and are not paid for or promoted by any business or entity.
Special thanks to Patrick Woods for editing this article.
