Can you turn that noise down, please?
One of the most troublesome aspects of the behavioral sciences is the bountiful presence of noise (Gelman, 2017). By noise, I mean the variability that fogs our ability to tell if changes in the independent variable had any effect on the dependent variable. Humans are all so incredibly different from one another in their characteristics and their behaviors; there are so many variables (that we admittedly cannot control) at play, and it’s frankly hard to keep track of them sometimes. Even the most seasoned researchers may not control for some random by mistake.
Everyone is so different from one another, in fact, that any one person is actually pretty distinct from the average person. Ironically, once there are a few variables in play, it’s quite rare for one person to be truly average. They may have average levels of one characteristic or another, but not all at once.
But why does this matter? It’s relevant because the behavioral sciences employ the use of statistical analyses that rely heavily on the comparisons of means, or averages. The result is few statistical values are suddenly representing a wide variety of human characteristics. There’s a lot of room for noise in these statistical representations of people, who come in all shapes and sizes.
There are many people who believe that our system of statistics in behavioral sciences is broken, partially due to the presence of so much noise (Gelman, 2017), but there is a certain amount of pushback from those folks who have found success in the current (broken) system, i.e., Susan Fiske & Co. (Gelman, 2017). There is plenty to say on the topic, but I am still gathering my own opinions and won’t discuss it any further here.
What I am interested in, however, is not only trying to minimize the noise present in research, but also reduce the severity of negative repercussions of that noise. While noise and error can never be entirely eradicated from science, research, when done correctly, does its best to control for anything that is controllable. I believe the goal is to limit the reducible error (James, 2013), but it may be that some scientists have lost the motivation or ability to do all in their power to do so.
As with much writing in both academic literature and in informal writing on the Internet, many people have identified these problems before and there is an increasingly open dialogue accordingly. This has become a bit of a pet peeve of mine; the identification of a problem, with no proposed solution. The growth of an idea starts with an open dialogue, informing others and persuading them to agree with your perspective. There is a certain point, however, when simply talking about it isn’t as helpful as actually doing something. No longer! I would like to take this opportunity to propose action.
In order to limit the scope here, I’d like to focus on what any given academic program can do to help decrease noise in research and reduce the negative effects of possibly lazy, incompetent, or rushed research.
Disclaimer: I have much to learn. By no means am I making faultless suggestions. Nevertheless, I hope to spur some thought in our readers.
Here are a couple ideas to consider:
· Incentivize replication studies
Replication studies are not common. Why should they be? Why on earth would you wake up one day and decide, “Hey, I’m going to do something that’s already been done!”? Encouraging more replication studies and incentivizing them may lead to a higher probability that any given research project gets a second look.
Not only is noise not eliminated by replication studies, but replications are equally as susceptible to noise as original trials. Despite this glaring issue, replications can be beneficial because it helps to keep researchers honest (Gelman, 2017). When your work is expected to endure closer scrutiny due to replications, you’re guaranteed to put in more effort to ensure everything is done the right way. No shortcuts and no silly mistakes. Yes, I know that shortcuts and carelessness are not the sole cause of noise in research, but limiting those certainly can’t hurt.
· Foster an environment where pride isn’t an issue and questions can be asked
From personal experience, I’m not sure how common this is, but I have heard this is a problem in some program cultures. Especially for graduate students and perhaps even faculty at times, there are times when “asking a stupid question” is possible. “Asking a stupid question” should never be possible. Researchers, whether they are students or teachers, fail to ask questions if they are unsure of some detail in their methodology or procedure. An academic program that strives to foster strong behavioral science (or any, for that matter) research must foster an environment in which all community members are encouraged to ask questions and feel safe doing their work. This goes both ways, too. Those who are answering questions should never shame another for asking something silly.
· Target Doctoral candidates and tenure candidates
In is my opinion that Doctoral students and tenure candidates are at the highest risk for carelessness. While this demographic is certainly more advanced than most students, PhD and tenure seekers are under the most pressure to generate publications. They are evaluated heavily based on the volume of work produced. While it is likely a faculty member seeking tenure is experienced enough to know when certain tweaks may be necessary to create more control in the experiment, cutting a corner may be the answer due to time, resource, or motivational restraints. Perhaps require replication studies for these folks? I’m sure there are plenty of undergrads or first year Masters students who would love to get lab experience…
· Allow social media to criticize, too
A certain APS hotshot recently bashed informal research critics on social media, but Andrew Gelman (2016) did a nice job of countering that hotshot’s argument (see Gelman’s blogpost here: http://andrewgelman.com/2016/09/21/what-has-happened-down-here-is-the-winds-have-changed/). The peer review process in academic literature is sacred, I get it. Peer review is the immune system of science. The problem is, it isn’t working (Gelman, 2016).
Social media and the Internet in general may not be the official peer review process that traditional academics are accustomed to, but the Internet, from YouTube troll to fellow researchers, is famously effective at facilitating mistake identification. Maybe not everyone likes it, but it’s sure as hell better than nothing. Science is all about questioning what you see! Social media could be a way to help spot mistakes that have been made and created more noise than was ideal.
As a last note, suggesting something like “requiring replication studies in tenure candidates” is admittedly naïve. Something like that would require an alarming amount of resources. But, you never know. It’s difficult to pull off on a small scale, even harder large scale, but the payoff to possibly solving these issues could mend a broken system.
Gelman, A. (2016, September 21). What has happened down here is the winds have changed [Blog post]. Retrieved from http://andrewgelman.com/2016/09/21/what-has-happened-down-here-is-the-winds-have-changed/
Gelman, A. (2017, February 11). Measurement error and the replication crisis [Blog post]. Retrieved from http://andrewgelman.com/2017/02/11/measurement-error-replication-crisis/
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 6). New York: springer.
