It’s not a replication crisis: It’s an innovation opportunity
This is the original version of the article, posted for transparency. Please read the updated version here.
It takes me a few moments to recognise the name. I’ve arrived early at the National Health and Medical Research Council’s Research Translation Symposium at the University of Sydney and I’m sitting, coffee in hand, planning my route through the day’s program. A gentleman, quietly spoken, in a sharp blue jacket, asks to share my table. His conference badge identifies him as Glenn Begley. We’re shaking hands when I remember his significance.
Six years ago, Begley published an article in the journal Nature that was, in hindsight, a seminal moment in science. In three tersely worded pages, it ushered in what’s become known as the “replication crisis” — a growing recognition that many published scientific findings cannot be reproduced by independent scientists and may, therefore, be untrue. There’s a rueful smile when I mention this, a slight rolling of the eyes. But he is soon telling me the back story.
It begins in 2002 when Begley left Australia for California to become Vice President of Hematology and Oncology at biopharmaceutical multinational Amgen. He’d made his name as a researcher at Melbourne’s Walter and Eliza Hall Institute. Amongst other achievements, his team was the first to discover human G-CSF, a protein found in bone marrow that can be used to hasten bone-marrow recovery after chemotherapy. The move from academic research to industry was, he tells me, a natural progression. “I saw it as an opportunity to have a real impact for patients.”
Begley soon put his new team to work, scouring the scientific journals for promising findings that might ultimately lead to a new treatment. For each finding, the first step was always replication. The Amgen scientists would carefully follow the methods described in the journal article to see if they could achieve the same result. If they failed, they would contact the original researchers. “Sometimes there are tricks to an experiment that are not fully described in the papers,” Begley explains. If they still couldn’t replicate the finding, they would arrange to visit the original researchers and observe them attempting to replicate their own experiments in their own labs. When that also failed, they would quietly abandon the project and move on to the next potential lead.
The turning point, Begley recalls, came one year at the American Association for Cancer Research conference where cancer scientist Lee Ellis accused Amgen and other industry players of failing patients by making only incremental advances in treatment. “Lee was right,” Begley says. “But we weren’t doing it deliberately. We weren’t setting out to develop drugs that were only 2% better than existing drugs.” Stung by the criticism, he revisited the Amgen records, selecting 53 ‘landmark’ projects. “These were major, seminal findings,” he says. “The most famous labs, the most famous researchers.” The records show that 47 of those projects had been discontinued because the original finding could not be replicated. “I went back to Lee Ellis,” Begley tells me, “and I said, “This is part of the problem.””
Co-written with Ellis, Begley’s Nature article made an immediate splash. But they didn’t get the response they were hoping for. “At conferences people would get in my face,” he tells me. “They would say Amgen scientists were hopeless, stupid, incompetent and so on without recognising that mostly it was the original investigators themselves who could not reproduce their own work. It was very unpleasant.” Then there was the hate mail. “The emails were so offensive that I just deleted them,” he says. “I never showed my wife.”
I ask Begley what happened to those 47 studies that didn’t replicate. Has the scientific community been notified? He explains that, in order to gain access to the labs, Amgen had to sign confidentiality agreements. It meant that only the original scientists could disclose the replication failures. To his knowledge, none have done so. Begley notes, however, that one of the 47 papers was retracted recently, amid accusations of data fabrication. “The work was groundbreaking,” he tells me, “and so we visited the lab on several occasions. This was in 2004, 2006. But they were unable to repeat their experiments for us.” He sighs. It’s clearly something he’s wrestled with over the years. “There are several tragedies,” he continues. “The tragedy for [the head scientist] as an individual. To finish your career like that, it’s so sad. The fact that so many other studies were building on this work, that’s a tragedy. The fact that the world wasted 10 or 12 years going down a dead end that could have been identified earlier.”
After Amgen, Begley continued working for biotech companies in the States. He presented to the Obama White House on scientific reproducibility. But in 2017 he was enticed back to Australia to lead BioCurate, a joint initiative from Melbourne and Monash Universities. The company’s aim, Begley explains, is to improve success rates in translating basic research into commercially viable products that reach patients and consumers. He describes his role, however, as “managing failure”. The harsh reality, he says, is that most projects won’t reach market. And while there are many things that can be done to avoid failure, it’s also an inevitable part of the scientific process. “Sometimes, we just don’t understand biology deeply enough,” he says. “That’s a perfectly acceptable reason for failure.”
The problem, Begley tells me, is that we load scientific results with emotional terms. “If the result is the one we like, we call it a positive study. If it’s one we don’t like, we call it a negative study. But the only truly failed study is one that’s uninterpretable. What we’re trying to do in this business is improve human health. A negative study, if its well done, might tell us that that whole area of research should stop. It’s a really valuable contribution. You’re just not going to make money out of it.”
Begley leans forward, adjusts his glasses. “I sometimes think that people deliberately try and misunderstand what I’m saying. I’m not saying that experiments shouldn’t fail. But an experiment that actually failed shouldn’t be presented as a success.” The crux of the problem, he argues, is academia’s focus on what he calls “sexy results”. “The incentives are to get a paper published in whatever journal so that I become famous, so that I get my next grant, so that I get elected to the academy. That’s the cycle that is perpetuated. That’s fundamentally the system that we have in academia. We’re talking about human behaviour. You provide people with an incentive and they’ll try and achieve what is required to get the reward. It’s like rats in a cage.”
“It’s pretty depressing,” I say. Begley counters immediately. “I’m not depressed,” he replies. “I’m actually incredibly optimistic about science. The advances that we’ve made in the last decades were unimaginable when I was in medical school. We’ve got treatments for rheumatoid arthritis, ankylosing spondylitis [a form of arthritis affecting the spine], inflammatory bowel disease, cancer, hepatitis C.” He shakes his head incredulously. “The world now has the opportunity to eradicate hepatitis C. Unimaginable! So one can’t be negative about science. And, if you go more broadly, the decrease in infant mortality, the increase in longevity both in western countries and in third world countries. Phenomenal! So you can’t be depressed.
“What you can say,” he continues, “is that we could do better. We’ve made enormous advances but we’ve wasted a lot of money. In the US the estimate is that it’s costing $28 billion a year for sloppy science. So there is a lot of lost opportunity. But the advances we’ve made are incredible. You’ve got to hold that in tension. That’s why I’ve never described this as a replication crisis. I’ve never used that word. It’s an opportunity. Because if we can take the funding from the lazy scientists and give it to the really good scientists, we’ll do even better. It really is an innovation opportunity.”
Around us, the venue is filling up. Delegates are queueing for coffee. Others are pinning their research posters to the felt-covered boards that line the walls. They have titles such as “Reducing waste by choosing the right systematic review type” and “Breaking down barriers to using research to guide clinical practice”. The program includes sessions on working with stake-holders to set research agendas, improving data sharing and access to scientific publications, and communicating research to patients and health professionals. Later this morning there’s a panel on “Reimagining Failure”. And though he disapproves of the term, Begley is one of four speakers addressing “The Replication Crisis”. Signs, I suggest, that the problems he identified are finally being taken seriously. “It’s so gratifying,” Begley says, “to see how the mood has changed, how much the world has moved on in the last six years. It’s wonderful.”
