Hurry, Don’t Rush
The COVID Files #1
I want to tell you a story about cancer. I promise you it’s relevant.
Introduction
High-dose chemotherapy is brutal.
The side effects you can feel include diarrhea, vomiting, nerve damage, horrible nausea, hair loss, etc.
The side effects you can’t feel are in many ways a lot worse. The primary one is the suppression of your bone marrow from making new blood cells. Red blood cells, white blood cells, platelets, they all get smashed. Enough chemotherapy and you don’t have a functioning immune system.
And you need one of those.
In the late 80s, a South African doctor was very bullish on a particular cure for breast cancer—chemotherapy combined with an autologous bone marrow transplant.
This procedure is extremely nasty, but you can recognise the logic in it immediately:
(1) remove a patient’s bone marrow and preserve it. Bone marrow plays the lead role in making new blood cells of all types.
(2) administer high-dose chemotherapy.
(3) transplant the removed, functional bone marrow back into the same patient.
(Note: the above is extremely unpleasant to experience.)
Result? A functioning immune system after even colossal doses of chemotherapeutic agents. See, chemo stops cell division everywhere (cancer + everything else), and bone marrow depression is the central factor in how much chemo is possible.
So, remove what basically amounts to a back-up immune system, administer a tonne of drugs, and then transplant that back-up right back in.
Complicated? Yes. Multiple procedures? Also yes. But survival?
Apparently, dramatically improved. The results were remarkable. The doctor, Werner Bezwoda, became instantly famous. By the early 90s, he was speaking all over the world, an oncologist pop star.
Clinics set aside whole dedicated spaces to perform this procedure. The number of bone marrow transplants performed skyrocketed. And the procedure received massive publicity in the early 90s when a patient named Nelene Fox sued her HMO for refusing to cover the procedure under insurance.
Actually, it wasn’t just because Fox sued — A LOT of people sued their insurance for not covering what was still regarded as an experimental procedure. It was because she sued, the case was drawn out while she got sicker, and then died… and then her estate won. The whole case was framed as the battle of a sick woman against a heartless corporation that eventually killed her.
The above paper is from 1994, and covers the tension between insurers, doctors, and patients well… and it contains a crucial phrase “no definitive evidence that it prolongs survival”. We’ll get to that.
What it doesn’t convey is the historical context. Cancer is scary now, while the 5 year survival rate for breast cancer now is about 90%. Oncology has made a tremendous amount of progress. But in the late 80s/early 90s cancer was a lot scarier, and widely regarded as a death sentence.
So people would try anything. And that was the problem.
At the same time clinics were setting up transplant shops and people were paying tens of thousands of US dollars to have this procedure performed privately, researchers who were seeking to recruit people to test Bezwoda’s protocol had a problem — they couldn’t recruit anyone to be in their clinical trials to test it.
So the procedure wasn’t just untested, it also couldn’t be easily tested, because what patient enrolls in a clinical trial where you get assigned at random to a control group, where you wouldn’t get a live-saving procedure? Do you want to be the dead data point which allows someone else to be the live one?
Those trials struggled on for years with their low recruitment rates and compounding frustration, but eventually concluded: the procedure didn’t work. No improvement in survival rates.
So a team went to South Africa to inspect Bezwoda’s work. They might have missed something. Where did his amazing figures come from?
Answer: from thin air. The trials were a sham. The treatment was appalling. The protocols were unregistered. The participants were most likely dead. The records were most likely non-existent.
The aftermath is not important.
Right now, I want you to hold all of the above in your mind:
- the dire need in the face of impending danger
- a strong, bold, risky treatment presented by a mercurial saviour
- the rush to use, and the overconfidence born from initial work
- the morphing of the issue into something larger and a lot messier when it hits the public domain
- the difficulties in performing testing because demand and access completely outstripped the ability to provide a controlled testing platform
- and the eventual disappointment, because of the unreliability of the early work
… because this is what I am afraid is happening right now with hydroxychloroquine (HCQ).
During the present COVID-19 omnishambles, I am afraid of us doing the above all over again, but faster. The Bezwoda case played out over more than a decade, the above has taken about a month.
And a far greater associated worry: we’ll do it elsewhere, as well. Other drugs. Other miracles. Other procedures.
I hope I’m wrong, but evidence is kind enough to leave clues. Let’s treat HCQ as a case study for a Bezwoda-style disappointment, and see if I’m full of it.
Onwards.
Hurry Don’t Rush
The following is a heavily shortened summary of the scientific evidence for HCQ as a frontline treatment for COVID right now. The situation is ridiculously fluid, and has been changing day to day as I wrote and edited this, so I hope it’s accurate at time of publication.
I invite you to read the attached links (warning: ~2e4 words minimum) and check my receipts.
- The first HCQ study has some extreme problems — the data is strangely censored, the ‘14-day follow up’ period is longer than the time between study approval and the manuscript appearing in public (and the data for the follow-up was unaccountably not made available), the control group and the intervention group were from different sites, the actual testing of the viral load was untrustworthy, the sample size is astonishingly small, and so on. Under normal circumstances, a journal would instantly desk-reject this paper. Instantly.
- The paper was published at predatory publisher speed (ie. almost instantly) at a journal where the authors have existing affiliations and/or editorial control. Peer review was cursory at best. Elsevier just released a statement which assures us that “the journal’s standard peer review process was followed in the publication of this paper”. Of course, the standard review process does not guarantee that a paper can get reviewed and accepted in 24 hours, so that seems… wrong.
- The results were then heavily promoted by a, uh, lawyer. He claimed a series of academic affiliations and authority which seems questionable at best. If he’s not a grifter, he certainly is wearing grifter’s clothing.
- The author/institute have produced a worrying series of previous papers containing duplicate and overlapping figures. I don’t know their reputation specific to their scientific field, but in any field that I do know, this makes me trust you a lot less.
- The International Society for Microbial Chemotherapy released a statement released a statement saying the first study did not meet the society’s expected standard. This is an extremely unusual step, for a society to censure a paper published in an affiliated journal, without the explicit input of the journal itself.
- The second HCQ study didn’t have a control group. What else needs to be added to that? If patients are treated for an illness and recover… they recover relative to what? By this standard, rubbing a penguin on your earlobes cures the common flu in 99% of cases, because people usually get better. With this kind of evidence, it makes as much sense to rush to the wet bit of the zoo if you have a sniffle.
- HCQ has some substantial side-effects. As it’s already being handed out indiscriminately, often to people with pre-existing conditions, this is a Serious Problem.
The above may not be what you were expecting, especially considering all the media fuss about HCQ. But that’s the most realistic take I can achieve, and I really don’t think I’m editorializing — those are the facts.
I’ve explained the above to a few non-scientists in the recent past, usually in more colourful language, and their reaction is uniformly one of confusion. How is this happening? How could you all let it happen? Why is it possible to do this?
The answer to this is straightforward: because it’s a symptom of the broader scientific environment right now. I would characterise that, unfortunately, like this:
[A] Some of the scientists you see putting work into public domain right now are producing good and valuable research at astonishing speed, which is helping us understand what’s happening and what we should do about it.
[B] Some of the scientists you see putting work into public domain right now are soulless attention-hungry weasels, and they are producing terrible research that is going to kill people. Less seriously, but still not good, they are riding their personal hobbyhorses into a landscape of hurried review papers about how their research in particular will clear up the post-COVID landscape and buy everyone a pony.
[C] A primary reason for [B] is that strategic conversations about how to capture COVID-related funding began weeks ago. Some scientists do research that is very well equipped to answer questions about a global viral infection (people who study globes and viruses, for instance). For others, the connection is more tenuous. For other still, the connection is an absolute fantasy, and an exercise in mental gymnastics. Regardless, they will all ask for money when it is available, and are positioning research now to be able to do so.
[D] The reasons for [C] are: (1) science in general is underfunded and even a lot of very successful scientists struggle for money, (2) a lot of scientific jobs and job security that is normally available is already disappearing, and a lot of regular scientific funding is very likely to shrink further in the near future because of the post-COVID economic environment, and everyone is worried about the short to medium term viability of their research, (3) universities and funding agencies have already started funding competitions to address COVID-related problems.
So, that’s what’s happening at a macro level. It’s also worth explaining the motives and context behind the above.
Heroes who hurry, weasels who rush
There are two reasons to try to rush scientific work through to publication in a crisis, and the one that all researchers would tell you is that we need the information right away, to help solve the problem. So, research is being produced, reviewed, and disseminated faster. We need to act now, we are told, and science is answering the call. There are people out there putting heroic hours into project that will seriously, actually, really help.
The reason that is less discussed is because the first people to publish COVID-related work get a lot of attention, and then they are in an ideal position to open their mouth first in front of the hosepipe of virus mitigation money that is already appearing from scientific funding bodies.
It is very hard to do novel scientific work accurately AND quickly AND completely, especially when human subjects are involved (and they often are in this case). This raises the straightforward reality that scientists are presently publishing speculative, incomplete, or partial results without normal due diligence.
This is not necessarily a problem. It is perfectly possible to publicly release partial information responsibly with cursory or zero formal peer review. This has been long discussed in the Open Science community, because it’s a conversation we’ve already had around preprints (a scientific paper released publicly before formal publication). I have three thousand unpublished words on this topic somewhere.
In any case, here’s what a responsible author of a pre-print or ‘study under duress’ can do to be more reasonable:
(1) Give realistic statements of the limitations within the paper. You should be well aware of what can go wrong with research. This should be trivial to include.
(2) When you make public comments, attempt to give perspective. Prepare for this because you know the interest will be immediate and extreme. Make a decision about what is and isn’t responsible to represent simply to the media and general public.
(3) Deliberately engage experts in the appropriate areas to assess your information publicly. If the work needs to be read by experts, find them yourself. In particular, put your work in front of people who might disagree with it.
(4) Admit that criticism of your work exists and then engage with it. You release a paper in advance of formal publication for DISCUSSION. Well, if that’s the case, get into the weeds and discuss it.
(5) Update your pre-print! It isn’t published yet. You can do whatever you like to it.
My present impression is that the exact opposite of this is happening right now. I think it’s more true to say we see a rush for positioning: papers released with breathless promotion, results which are overconfident, oversold, or in some cases almost incoherent, and a certain unwillingness to engage in public discussion, especially by authors. That does not preclude them, of course, from hiding behind the trope of “just asking questions” or “raising issues”. Just asking questions is fine, until you stick your fingers in your ears when you get savage answers.
Conclusion
Make no mistake: we need science right now. There’s a global problem whose solution lies squarely in the domain of building empirical knowledge about the natural world. This comes with something of a gold rush, both for money and for profile.
But the historical precedent for bad scientific work wasting millions of dollars and hours is strong. The specific precedents for work that met an immediate need and looked cool, but ended up being revealed as desperately overhyped, are also substantial. And right now, instead of bad science just slowing down progress, it can fly straight to the heart of global policymaking on the run. This makes it not some milquetoast term we’d normally use, like “problematic”, but rather dangerous.
I hope this doesn’t happen, but it already happens when the world is spinning on its normal axis. Right now, twisting out of control, it has the potential to both happen more and stack bodies.
