What should we replicate? Things that will end the replication crisis
If you didn’t know (most scientists let alone citizens don’t, so you’re getting in on the ground floor!), there’s a replication crisis in science (if know all about this, skip down to “So now you know: What to do now?) (or read on, it’s a fun story, with cool links) :-)
TLDR? Influence ÷ evidence.
Most of what is published is false. Not in the “we never know the truth” sense but in the devastating, Pons and Fleischmann cold fusion sense: Run again, the experiments don’t deliver what the original authors claimed.
Yip: Run an average study you read about in a high-quality, peer-reviewed journal and it is unlikely you will get results that differ from chance.
You can see this is bad. We do science for two important reasons; The first is simple curiosity: who isn’t awed, knowing that the Sun is a 4.6 billion year-old fusion reactor? The second reason is more practical: While invention can come from other places than science, there’s “nothing as practical as a good theory”.
It was creative tenacious people following the rules of science who gave us the transistor. And the secret of life, and the ability to replicate DNA, and amplify light. It’s science that gave us the Polio vaccine, Anaesthetics (invented not 1,000 yards from where I am penning this essay)… The list is wonderful: these are the things have made life phenomenally much better. Why?
Because they work!
Something that looks like science but lacks this property of working is a cargo-cult: It’s cargo-cult science. We can imagine how harmful this is with a simple thought experiment.
Imagine Jim Watson in a replication-crisis world. Imagine if instead of figuring out how two anti-parallel strands of sugar topped by just 4 kinds of amino acid and wound in a spiral, could (when read-off by similarly beautiful molecular machinery) code for life itself.
Oh, and it contained the code to copy itself… This was simply incredible, productive, transformative and agency-raising for humanity on all levels.
Now, imagine a darker, world. A world of dim candles not electric lights. You’re hungry, and cold because science is broken. And at your bedside, “anti-Jim” conjures up a night-time story for you: “Imagine” he says, “a beautiful balloon”:
Imagine instead of pipe-dreams of invisible “X-Ray cystallography” (only the crazy old lady down the road believed in that fortune teller’s myth) — Imagine he say our cells are programmed by how you pose your body “embodied cognition”. He launches a TED talk, watched millions of times at dull fire places around the village. Life, he exclams, is carried by “social-priming force”. You don’t know about it because it’s unconscious. But he has proof – “incontrovertible evidence”. “Absolutely, we know he says” and tells you he and his students (also on the witch-doctor market now) have shown this “dozens of times”. Newly-printed research dollars lurch in this direction: it’s the secret of a happy life. Hiring falls almost entirely in the lap of this new dream team. For decades.
Until:
“And like that… he’s gone” The bubble bursts. None of those claims were true. The first story-assistant probably fudged some of the data, and the rest of the dream-tellers fell into line because it was such a good story and the old stories were “boring”.
Why does it matter?
Unless we are doing science as a game to pay off our mortgages, get rich with book contracts, or stump for our favourite political view, then it’s easy to see that bad science is really bad.
Bad science is wrong ideas. It diverts capital and precious time. It fails to work, and may even be harmful.
In education, bad science is children not learning and spending their lives poor as a result. In medicine, it’s pharmaceuticals treating the symptom but not the cause, and surgeries performed years after better ways are known.
So now you know: What to do now?
The only way we know to stop bad science is to stop each of us from doing it. And people are thinking about exactly how best to make this happen. Like here.
Doing this takes:
- Clear methods for demonstrating when someone’s work is bad.
- Lots of these demonstrations.
- Methods to communicate these results. Not just publish them (though that’s a massive barrier), but to market them so they enter the same minds that watch TED, and read the newspapers. And to market the bigger idea of asking “Where’s the beef?” So the producers, journalists and high-order bits in the communication world: The Neil deGrasse Tysons who can talk direct to millions ask hard questions instead of falling for easy but ludicrous and unproven answers.
Big job!
Worse, like the Borg or the Empire…
“Bad Science: The empire is fighting back”
Creators of bad science don’t want to lose their reputations, book-contracts, promotions, and prestige (or ability to get students lifetime tenure, get grants, and go on TV). So they say in turns “It IS true!” and “You can’t prove the null!”. “You’re the failure!” and “You’re bullying me: You’re a terrorist!”. They mail heads of your department, editors, colleagues, and the university demanding you are sanctioned for even discussing them. They threaten to sue.
Even worse, these bad-eggs are making the wonderful, optimistic, humble, and heroic Brian Nosek look pensive, which is just plain wrong! (seriously!)
Evidence and marketing needed now!
Brent Roberts made a great suggestion: Set aside a pool of money specifically for replicating studies. That’s a great start. Let’s put 20% into replication. Right across the board. Have replication grants, which only say “I’ll replicate studies from…” Yes: from what list? How shall we decide?
I have a very simple metric:
Influence/evidence.
Influence divided by evidence. Top of the list are papers that are getting cited a hundreds of times but have yet to be replicated even once: That’s a big flashing red warning sign!
People love your work enough to cite it over 700 times, but not one of them had any need to do any work that allowed them to test it? How likely is that? Turns out this is not uncommon, after a year-long battle we’ve published exactly that kind of replication just this year (2016). Now that’s a whole ‘nuther story :-).
Others near the top of the list are pretty much any paper authors have lobbied their PR department to make press-releases on, or managed to get media friends to report on in the newspaper. Top of the list for people writing their own puff pieces in venues like the New York Times: now, sadly, too cash poor to write such complex material and reduced to often recycling press releases like other outlets. And 10-x the urgency for papers reported prior to publication: There’s nearly always something fishy there.
Craving the public ear? You get replicated.
How can we do this? What’s the light-weight rapid turn-around funding tool?
There should be a “fast-track”: See a report you don’t believe? Get funding for a direct replication in a month. I would organize this in a system where any scientist with 10 pubmed dois (“a good reputation” for those in the biz) can suggest papers in need of replication. An automatic ranking system (perhaps supplemented by allowing scientists to vote up papers) would list up the suggestions, and anything that crosses a certain threshold, the funder automatically issues a tender for replication. Who to fund? Easy: Like on e-bay, the lowest-bidding reliable vendor wins the contract at the second-bidders price.
Few will risk their reputation by running a replication which, itself, later fails to replicate. And if they do, or if they fail to deliver on time, we lower their reputation.
In social science, most replications should be turned around from suggestion to vote, to tender, to analysed data in about 3-months. Get newspapers and TV stations to sign up to a “replication contract”, where they promise to run a 2-minute piece on any story they run that fails to replicate.
They’ll love it! Just like students (the same ones who love hearing lectures on pulp-psychology) love the intrigue, detective work, and scandal of failures to replicate.
What else should we prioritise?
Anything discussed on TED/equivalent (Incredibly, studies that form the heart of many TED talks are unreplicated, unreplicable, or not even published… ). All of these are must-replicate scenarios.
What else?
If there’s money left over, then let’s just fund replication for everything in order of how much it is being cited. And let’s keep running replications until all these papers either have a solid body of independent replication support, as well as evidence for lack of publication bias). Or until they are shown to be false.
So I think Brent’s onto a winner.
Shouldn’t we just ignore all this, and focus on public media attention?
Jim Coyne suggests a strategy of ignoring the preposterous: We can’t afford the time of skilled scientists replicating a hydra-headed stream of noise. The key instead, he suggests is social-media shaming.
I think this is both wrong and right: Wrong because we need to show that the science is bad. Not all of it is. What is wrong is not “obvious” — that was one of the key inventjoins of science: That we can not know the right answer But we can learn. So it’s just unscientific to ignore work.. He’s right too, though, because MARKETING AND COMMUNICATION MATTER. But this will be SOOO much more powerful with peer-reviewed failures to replicate that are easily accesssible. With automated publication bias and meta-analytic charts injected into wikipedia articles via template. With easily searchable systems for journalists on the track of the latest good story.
It’s great we disagree: open society is part of the science invention itself. Smart, thoughtful people propose ideas: One day Jim’s right, the next Brent, and one day me. But each and every day, everybody wins: That’s the beauty of the wider social web and the magic that is our precious invention: science itself.
(all those links ↑ are great books, BTW!)
Snake Oil
The trouble, IMHO, with Jim’s idea is that what shaming snake-oil on twitter as being snake-oil, is that it won’t combat what is essentially an endless supply of airport books, and unscrupulous folk starting businesses doing brain training. Or offering “cures” for dyslexia based on balancing balls, special glasses, or a wealth of other snake oil (including actual fish oil).
It’s not enough just to say “that’s silly” because having a prestigious position in a prestigious institution is one of the things that both elevates and insulates these claims from scrutiny, while propelling them into the public mind. Many of most dubious salesmen and women now work in our most prestigious universities. We’ve been too polite to check results, too timid to publish failures, and too cowed by threats. Into that vacuum, more and more outrageous claims have been launched and lead to prestigious hires: Who wouldn’t hire the shiny new student with “effects in education greater than anything anyone’s ever shown before…” you can hear the hiring committee gleaming.
So I think we should very much not ignore the snake oil. Instead, I think we should specifically target it. It will be the easiest to reveal as false, and that, linked prominently to the original work, preferably published in the journals hosting the bad science, will halt most funders in their tracks, and give otherwise happy writers before citing bad work which publisher’s automatic softwae will automatically flag as being suspicious.
This will have an immediate positive spin-off: Once students from failed labs start failing to get great job offers, or indeed any job offers, the narrative-making business will simply dry up.
So: think of the children, and ask yourself “Could I start a replication today?”
The answer is in your head: It’s “yes: I can!”.
Now: What paper is it you see at the top of the “influence ÷ evidence” hierarchy in your field?
And reach out: there are thousands of us willing to help!
