Slaves to significance
Next month is a significant month for me: I finally finish all of my coursework for the program as well as complete my thesis proposal. My thesis defense will be later in the summer, and them I’m done; I will have finished the race. I know that even if I don’t find significance in my results, as long as I have listened to the criticisms of my committee and made hard strides to facilitate accurate research, I will have learned a great amount and will graduate. While that alone is worth the price of admission, still there would be this looming sense of failure over the lack of significance: Was my research actually good? Did I actually make any contributions? How can I expect success if I can’t even do this correctly?
While these worries may seem a bit extreme, I feel they accurately reflect the greater picture of the now-commonplace obsession with the p-value, which is seemingly no longer even a measure of data, but a measure of personal and professional worth. This bastardization of the p-value is what this week’s readings address, all of which approach the topic with an inherent disappointment and frustration that is to be admired, as it seems the authors truly understand the gravity of the issue at hand.
Greenland et al. (2016) begin their work by noting that some journals are now outright discouraging analyses that rely heavily on the p-value. Personally, I feel there is probably no greater indication of a serious problem than if you have journals that are turning their backs on the now fundamental statistical value. Why and how there are even researchers and scholars arguing against this is beyond me. Regardless, before the authors move into their 25 misconceptions about the p-value, they take good time to note the heavy assumptions and difficulty involved in the overall statistical model, noting how its scope is much broader and significant than is often considered. This is great to hear because it emphasizes understanding of the data as a whole, a point I harped on in my previous post and a lesson learned earlier in the semester when we read a chapter on exploratory data analysis by Behrens and Yu (2012). We need to be equally flexible and rigorous with our data.
The misconception point that best emphasizes both the importance of understanding all of your data and the silliness of over-reliance on the p-value is #6. The authors note that if a study yields a p-value greater than .05, the common assumption is that there is no evidence or association of an effect (Greenland et al., 2016). But this is deeply wrong. Consider this framing of the issue: I’m going to completely ignore the rest of my data and forego the possibility that an effect exists just because my p-value is greater than .05. While that framing may be overly-reductive, making anywhere close to that assumption is nonsense. Your data is bigger and more complex than your p-value.
The next two readings are both blog posts from Andrew Galman. The first is an interview with Galman that notes the serious effect of noise on our research. “ Noise is random error that interferes with our ability to observe a clear signal” (Galman, 2017). Galman notes that significant p-values can be the result of this noise. Yes, our precious and supernatural p-value can actually just be the result of something that is entirely irrelevant to our interests and research. Once again, this feeds back into the need to understand all of your data. Everything needs context!
A final note in Galman’s interview is discussion of the benefits and downsides of replication. He mentions that he would gladly support replication if all it meant is that it kept researchers honest. I think this is equally important and sad. Replication is underappreciated and likely scoffed at by some, so it’s important that it be considered, but I think it’s sad that it has to be established as a baseline system of checks and balances instead of the norm for good research.
The final reading is Galman’s critique and review of research and academia, framed beautifully as the story of a flood that has washed good, honest research away, taking one too many people with it. He starts immediately by noting the internal going-ons of the psychology realm: the smear tactics, looking out only for one’s career, etc., tactics we may commonly think are reserved for politics and business. And while he doesn’t focus on this, it speaks to the totalitarian regime of significance and how much we are bound by it. Some of us are unfortunately in a position where significance in some sense may determine our livelihood (Galman, 2016).
As he moves forward, the frustration becomes more evident. There are clear and evident errors in published papers that are not handled. There are major issues with the current state of publication. There are notable scholars and researchers failing to take responsibility for their work and refusing to hear criticism. Some claim that criticism should only happen in private, and others claim that you are required to believe the higher-level research. Honestly, it sounds exactly like a corrupt government looking out for their own interests, one that might cry “fake news” when they are confronted with something they don’t like. It’s entirely disappointing, and perhaps the most heartbreaking line in any of these readings is “When the authors protest that none of the errors really matter, it makes you realize that, in these projects, the data hardly matter at all” (Galman, 2016).
We do not care about the data, we care only about the results, and we bend to the will of the p-value that gives us those results. Because at the end of the day, doing good research doesn’t make you a good researcher.
References
Behrens, J. T., & Yu, C. (2012). Exploratory Data Analysis. In J. A. Schinka, W. F. Velicer, & I. B. Weiner (Eds.), Handbook of Psychology: Volume 2 Research Methods in Psychology (pp. 33–60). Hoboken, NJ: Wiley.
Gelman, A. (2016, September 21). What has happened down here is the winds have changed [Blog post]. Retrieved from http://andrewgelman.com/2016/09/21/what-has-happened-down-here-is-the-winds-have-changed/
Gelman, A. (2017, February 11). Measurement error and the replication crisis [Blog post]. Retrieved from http://andrewgelman.com/2017/02/11/measurement-error-replication-crisis/
Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical tests, P. European journal of epidemiology, 31(4), 337–350.
