Better incentives, Better science
The Open Science Framework does methods properly, but the replicability crisis isn’t due to poor methods.
Go read Brent Robert’s response to the replicability project. It’s super good and lays out a new platinum standard for scientific reliability. While you’re at it, consider the AMA with Brian Nosek about the open science framework (OSF) as well.
- I am immensely sympathetic to the notion that science, as currently performed, particularly in the social and biological sciences, is not reliable.
- I think that this poor reliability is related to the mediocre methods and statistics currently in use.
- Projects like the replicability project provide strong empirical evidence that science isn’t living up to an acceptable level of reliability.
- Most of the responses to the replicability project are pathetic. “Hidden mediators” are not to blame. And that’s me being nice because if they are to blame the findings are even weaker than advertised!
But as someone who is not yet a tenured professor. I think that the OSF approach isn’t for me, and doesn’t address the incentives to do bad science. I don’t think poor reliability is caused by poor methods. I think both bad science and bad methods are caused by bad incentives for scientists, especially young, untenured scientists. I had hoped that the replicability project would be an opportunity to name and address these bad incentives.
I fear the open science framework will just be one more barrier for young investigators science. In its current form, it doesn’t attack the sources of poor reliability. It might even render science less reliable.
My Grandfather and Jack Anderson
This is my grandfather, Fred Kummerow. He is a nutritional biochemist at the University of Illinois and for much of his life he researched the effects of trans fats on heart disease.
His first publication on the dangers of trans fat was published in Science in 1957. He published about 50 additional experiments replicating this core finding.
It’s a reliable effect. He was funded by the NIH until 1979.
This is Jack Anderson. He was a satirist and investigative journalist. He’s most famous for his takedowns of Nixon during Watergate.
In 1979, he satirised my grandfather. My grandfather had testified to the American Heart Association that trans fat was a larger danger than cholesterol.
Turns out my grandfather had one graduate student, funded by the American Egg Board to the tune of $12k/year. Anderson got to make a hilarious “egg on his face” reference.
After this article the NIH chose not renew my grandfather’s $400k/year grant. At least he still had his egg money.
This is Unilever. They developed a trans fat free method for making shortening in 2007. They supported research like my grandfather’s to get people to buy their trans fat free products instead of their competitors.
In 2014, the FDA banned trans fat.
Trans fat will kill you, and I know a guy who knew that in 1957. But his careful methods and consistent replication wasn’t enough. Jack Anderson wasn’t trying to hurt anyone and Unilever was only trying to help themselves and yet…here we are.
Science takes place in a Madisonian system, where competing forces push us from all directions. The good guys don’t always win. The powerful guys push hard. The direction science moves depends on the sum of the forces on it. A well meaning push in the right direction doesn’t always end up moving science that way. The reliability project and the open science foundation have recently come into a bit of power and public influence. So far, they’ve pushed for greater reliability and more careful methods.
But they’ve yet to ask “Why are scientists using poor methods?”, “What are the forces that drive reliability down?”, and “Why were past methods more reliable?” The surge in retraction is relatively new. There wasn’t an OSF in 1957, and yet, somehow my grandfather was able to publish a reliable effect.
My grandfather, satirized in the national press though he was, had no fear of being fired. He had tenure by the time he was my age, and was publishing his science without fear of being fired, while my job is year to year, funding permitting. Although I’ve had the same education and published a similar number of papers to my grandfather, I’m still in a science waiting room. I need to hit a home run. Until then, I live in fear that some of the work I’m associated with might get satirized.
Compared to my grandfather’s time, job security is rare among the people who actually perform the science. Most scientists today get their groceries from new, sexy findings. Cutting-edge methods are often the least reliable, but the most expensive, encouraging scientists to write grants that use these techniques instead of well-validated, inexpensive ones. Public interest in science has turned investigators into scientific evangelists who drum up public support to enhance their fundraising ability.
I don’t see how open science framework’s pre-registration addresses any of the interrelated forces that got us here in the first place: greater public arbitration, academic feudalism, and resource dilemmas.
Public arbitration
A Major US university is choosing which candidate to hire:
Candidate A: Has conducted 1o pre-registered OSF studies. All of them show that previous results (some by scientists at this university) fail to replicate. They have published these results in the few, middle-tier journals that are willing to accept them.
Candidate B: Has 1 study in Science that went viral on Buzzfeed.
Every university in the country will hire candidate B.
Every university has to hire candidate B. Universities are in the public relations business. They need to attract students, seduce donors, and carry water for legislators. If they don’t make their research sexy, they will soon have no research at all. This goes double for corporations, national labs, and private patrons.
Greater public engagement in science means that Buzzfeed and reddit get to vet scientific findings. Buzzfeed and reddit want sweaty t-shirts, and cures for cancer.
The same-sex-marriage study was published because of it’s political, publicly appealing content. It was investigated for the same reason. When the investigation demonstrated the data was faked, the article was retracted. This retraction attracted attention because of the political content! The public is part of the peer review process, and they don’t care about methods.
Nor should they! It is unreasonable to hold the public to a standard of evidence and methodological care that expert reviewers in the field routinely fail to meet.
Preregistration does nothing to change this. Bad studies about underpants will still get attention because the medias is paid by the click, not the fact.
This probably should be clear to OSF because after all:
The award for best example of using public appeals to gain leverage over an entrenched academic system goes to: The replicability project.
Oligarchy
One of Brent’s comments that I reacted very strongly to was this:
Don’t leave the replications to the young. Senior researchers, the ones with tenure, should be the front line of replication research — especially if it is their research that is not replicating. They are the ones who can suffer the reputational hits and not lose their paychecks. If we want the field to change quickly and effectively, the senior researchers must lead, not follow.
This framing seems to conceive of science as an ivory ranch house, with relatively flat power relations. This does not match my experience. I work in an ivory tower. Above me are the tenured PIs with long careers and strong reputations. Below me are the undergraduate researchers gathering data. At my level are a large pool of post-docs I compete with. We serve those above us, direct those below us, and stab those beside us.
I am currently at work on a large, OSF pre-registered study. The PIs on this project are good and honest types like my grandfather. They are trying to do the right thing.
But they aren’t risking their reputations! They’re risking mine!
If they get a cool result based on the pre-registration, they secure their reputations as insightful investigators. If no such result emerges, they still secure their reputations as hard nosed skeptics.
But the legion of RAs, graduate students, subject runners, and post-docs who gather and analyse the data need a result. These folks depend on this grant to pay their bills. These people are building their careers.
For many of us, this will be our only opportunity to do science in our lives.
So when Brent says tenured professors ought to be at the forefront of preregistration, he’s actually asking powerful people to risk other people’s reputations. Tenured professors can throw their minions away on pre-registered studies. Nearly everyone else is in the Darwinian phase of their career.
I like to use Game of Thrones references when I talk about academic feudalism. Department chairs are Lords of great houses. PIs are important banner-men. When these folks go to war, it’s the small folk who suffer.
Pre-registration doesn’t calm the competition at the bottom of the tower where dodgy methods proliferate. It makes the tower harder to climb.
The award for best new barrier to entry for young scientists: OSF pre-registration.
Resource Dilemmas
Competition for grants, fellowships, and attention is fierce, and has a chilling effect on new investigators. I don’t have time, attention, or money to undertake new research, let alone replicate old stuff.
Applying for money is an arduous process that takes a tremendous amount of time and will likely result in nothing. Last year I went to one of the SFN satellite conferences, where an NIMH representative told us in reverent tones that there was a special category of grants in which …*gasp!* 24% were funded! Holy frijoles! If I could write a grant in a month, in just four tries I could have a whole year of funding…for an undergraduate RA…not me.
Of course, that grant category wasn’t for replications. It was for new (expensive, unreliable) tools!
Not that I would bother. Applying for new money and jobs is a depressing experience. That competition is crammed with assistant professors one level up from me on the tower all of them red in tooth and claw.
I’m not a fund raiser or a politician. I didn’t get into neuroscience to try to understand the NIH funding schedule. I’ve been fortunate enough to be funded so far, but every minute I spend self-promoting, schmoozing, and chasing new projects (that will pay me to chase new projects) is a minute I spend away from the job I want to do.
It reminds me to be a cynic, and that science (like most professions), provides merit pay to its most self-promoting and narcissistic workers, not it’s best.
And yet:
The award for best use of scientific resources: The replicability project.
The replicability project was run on nights and weekends by honest people out of their own pockets. It was an important sacrifice. It produced one of the most important findings of this decade. The people who did it are heroes.
Why does NIH pay me and not them?
Please OSF, be Unilever
I have been hard on OSF and the replicability project. That’s because I think they have to succeed. They represent an absolutely vital counter-narrative to a modern scientism driven by social media, entrenched academics, and funding sources that prioritize big names and sexy stories.
Yet I feel that OSF is being Jack Anderson. Yes, they are holding science to a higher standard and dutifully revealing shenanigans. But they are ignoring the structural forces that create shenanigans in the first place.
My grandfather didn’t take money from an egg producer because he loved eggs and wanted to shill. He took it to fund a student. He took it so he wouldn’t have to write another grant. So he could lose his funding and not have to fire that kid. He played competing forces against each other to obtain a little more independence, which I hope helped him see the truth a little more clearly.
Jack Anderson saw the trees: my grandfather’s funding made him beholden to egg producers. But he missed the forest: scientists serve many masters. Investigators, patrons, and studies don’t exist in a vacuum. It’s the robust contest among perspectives that gets us closer to understand the truth.
When he lost his funding, my grandfather didn’t stop publishing. He found an oil man who would back his lab, and kept writing papers about trans fat. He published his most recent one at the age of 99. It is the longest con for the American Egg Board on record.
Right now the replicability project has tremendous clout and power to do good for science. They have chosen to use it to set up a rigorous system for filtering out cheating. They have not addressed the structural problems that help cheaters prosper.
Constructive recommendations
I’ll close by not being such a sourpuss. I said before that OSF was pretty close to my preferred model of science, and I’ll finish that thought here to be constructive. I actually think we’re close to scientific utopia, and that OSF is an important step on the way there. OSF is a platform that vets research with sunlight, it can also provide security to motivated discoverers.
- OSF instead of publication, End-to-end Peer-review.
The brilliant part of OSF and pre-reg is that the study is vetted when you design it, not after you’ve done your best to make it sexy. Yet currently, there’s a disincentive to use OSF because you’re essentially signing up for two reviews. Besides being a pain, what if your manuscript reviewers and pre-reviewers disagree?
Why have a separate manuscript review at all? Why not just plug PLOS, Frontiers, or an arXiv overlay journal right into the back end of OSF? You’ve already got independent peer reviewers vetting you each step of the way. You can change something or run additional participants. By passing preregistration, you guarantee a publication on the back end. What’s more, the entire process of interaction with a reviewer is transparent and public once the manuscript goes live.
It’s not like we have limited space. We don’t write on dead trees anymore. Reviews are not sent through the mail. There is never a reason to reject an outcome.
What’s more, dynamic peer-review feedback, is just a hop-skip and a jump from dynamic collaboration. Need an analysis from a world expert in a particular technique? Why not ping them on OSF? Want your paper to have an instant replication? Now you can welcome people running the same task instead of hiding from them at conventions.
And the public record of that dynamic collaboration on the OSF forum takes the place of resume-padding review articles. I think my ideal day would be synthesizing ideas in a forum that lets me start a test of my theories with a few clicks.
2. Break the link between funding and results.
It’s the money stupid. We are paying scientists for sexy results and they are delivering.
I have a four-year-old son and a job in science. If my funding next year depends on scientific results this year, I cannot possibly be an unbiased source. This is not a controversial idea. Tenure is supposed to shield scientists from worrying about groceries. But the vast majority of people doing science don’t have tenure. We cannot hold graduate students paid $25k/year with no job security to the same technical and ethical standards as individuals with a guaranteed upper-middle class job for life. Universities and governments have gotten the reliability they’ve paid for.
In addition, we cannot pay by the method. A survey costs $8 per hour. An MRI costs $600. Therefore: an MRI is intrinsically 75 times less reliable than a piece of paper, but also 75 times more enticing. Wrong, expensive, flashy methods are still wrong methods.
I’m currently on a grant that requires us to register with OSF. Good for them! But also, if we find a result, we get another year of funding. If we don’t we get fired. Do they seriously think OSF is going to “keep us honest?”
Funders are already using OSF as a bouncer. They’ve learned to buy the results they want, and now they plan to pay a little extra to get the reliability they want. Science does not work that way. Discoveries are not for sale.
I’m a radical. I think the best way to break the grip of institutions like journals is independent investigators. Ten year fellowships at a living wage for everyone. No overhead. Everyone pays research costs out of pocket. OSF could even be a good platform for scientists to form kickstarter-style consortiums for research that they feel is especially worthy but is difficult for an individual to afford. But I’d settle for OSF declining preregistration on studies with funding contingencies. It is not OSF’s purpose to compete on providing the “best” results.
It’s clear that OSF’s goal is to become a new platform for science. Instead of a Funder, University, Journal iron triangle, everything will flow through the open framework. That’s brilliant and you’ve got my vote, but I’m paid by the existing system.
It wouldn’t take much to change my mind! I just need a pipeline to publication, and funding security. I hope that the replicability project represents a wind blowing in this direction. I fear it’s just another wind in my face.
I don’t know if my grandfather was right to take the egg money. I’m certain Jack Anderson was just trying to do the right thing. I’m certain Unilever was just trying to make a buck.
But I like to imagine that a few questionable egg dollars saved a bunch of hearts.
And I hope a few questionable p-values can save a bunch of minds.
