The damage is already done: proposed improvements to research practices are the cure
If you’re against the reforms, kindly get out of the way and let others fix the mess
There are researchers doing their utmost to improve research practices in psychological science. They promote better publishing practices and open science. They advocate for improvements in; research design, statistical methods used, and greater statistical power. They even attempt to correct the scientific record. They are in the minority, but momentum is growing.
In principle, I don’t think that anybody has anything against improving research methods. Better methods lead to more informative results; it’s hard to argue otherwise. Most researchers are very good at pointing out how any given study could have been better designed to yield more informative results.
Yet, psychological science is in a state of flux. It often feels like a battle between advocates for improving research standards (read: open, transparent, reproducible, robust, etc) and those that prefer the status quo. It seems to me that a large part of this head-butting emerges from the unavoidable necessity of pointing out flaws in an existing system. Together, these so-called ‘revolutionaries’¹ have highlighted broken aspects of our field (massive publication bias, shoddy research practices, poor study design, low statistical power across the board, failure to validate measures or report their reliability, to name only a few).
Warning: I’m now going to talk to you like a child. A basic principle of science is that when something is wrong, or incorrect, we fix it and do better next time. So, pointing out flaws is a good thing for science as it helps us improve. Please correct me if I am being overly simplistic. Now, back to adult-talk.
Unfortunately, some believe that what these heroes are doing is wrong. Maybe they believe that being open about problems makes our science ‘look bad’. Maybe, within certain cliques of researchers, questioning the research/publishing strategy of a friend/senior researcher is considered taboo. Maybe, just maybe, some know that with robust research practices their own pet effect would not hold up to scrutiny. At worst, some seem to hold a genuine belief that exposing the need for improvement is in itself doing a disservice to academia. Perhaps they do not understand the scope of the problem; though that is no longer much of an excuse.
None of these are valid excuses for actively standing against proposed practices to improve psychological science. If you genuinely believe that correcting errors, preregistration and registered reports, open data, sharing analysis scripts, etc. are doing harm; you need to know that the harm is already done.
The harm is already done. The very practices you are defending (even if only by an omission to recognise the problem in the first instance) caused the problem in the first place. Poor methods, overblown conclusions, low power, publication bias, storytelling, HARKing, p-hacking, and so many other common practices are a disease. Exposing and correcting errors in the published literature, preregistration and registered reports, open data, and sharing analysis scripts to name a few initiatives; they are the cure.
What is worse; “we found problems and we are trying to fix them”, or “what problem, no problem here, these people saying things are wrong are trouble makers, everything is fine because I was successful”. I guarantee you that anyone without a vested interest in the current system (ECRs, undergrads, and the lay public) immediately finds it clear that fixing problems is far more positive than denying their existence.
Perhaps a reasonable excuse is that most do not ‘get’ the problem or it’s scope. That is fine, here is a link to a poster containing eight papers I hope will help you understand. The papers cover issues that need to be addressed and some measures to act as cures. Grab a mug of tea and have a read². Understanding the problems will hopefully make you understand our frustration at unnecessary roadblocks to progress. You may have no inclination to become an open science advocate, but please do acknowledge the benefits of improved practices and let others pave the way forward.
But, please do not assume that because these practices take more time, effort, or understanding than you are willing to put in that they are a bad thing. Please do not pretend that pushing for reforms is a case of ‘bullying’ certain researchers into actually conducting and reporting their research robustly. Finally, I urge you, please do not argue that these improved practices or “methodological terrorists” are doing harm. They are trying to fix the mess others have created.
By standing against progress you are actively supporting these crimes against psychological science and helping to poison our field’s reputation. If you cannot stand with us to make progress, at least do us the favour and get out of the way.
[1] While I disagree with this piece, it is where I first came across the “Revolutionaries” title (link)
[2] Totally a not-so-subtle reference to ReproducibiliTea
