Yet another copy/pasted Twitter thread!
In addition to being a professor, I also do freelance statistical consulting (my rent is too damn high). I thought it would be interesting to go over the way that people think about statistics in this hidden little corner of the internet.
To be specific, I consult on research and analysis design, and also do coding work. Most clients are small businesses or independent. A few bigger organizations, and the odd student rich enough to drop a few hundo on getting a prof to critique their term paper. Rarely, I’ll work with academics, too.
The first big thing you notice is that a lot of people tend to think of statistics as a tool that serves a function, rather than as a method of analysis.
What I mean by this is that it’s common to think of statistical analysis almost like an enterprise product. If I buy this “statistics” doodad, it will help me make my point. If it doesn’t do that, it serves no purpose.
It’s extremely common that I will get requests to “fix” the work if it doesn’t give them the result they want. I’ve had people demand refunds. Plenty of job ads that tell the worker the results they’re supposed to get. I don’t respond to those ones.
It’s tempting to think of this as fraudulent, but similar to the current discussion about the replication crisis in academia I think the jump to negative intent is not quite correct.
In the conversations I have with people, I think it’s more a deep misunderstanding of the purpose of statistics and empiricism. It’s extremely difficult to even get across the point of why “fixing the results” is a bad thing. And believe me, I’ve tried this conversation many times. They want good statistics and “best practices.” They just literally think that this means doing whatever gives you the “right” result.
In a related note, p-hacking is just an incredibly natural temptation. I’d say that a good 50% of the insignificant results I return get a response asking how they can be changed. The most common phrasing is asking how they can be “fixed.” There’s just no concept of that being bad. It would never occur to them that this might be the wrong thing to do.
I say “natural” from academic experience. I’ve never seen a statistics or econometrics class that teaches p-hacking as good. Many explicitly warn against it. But when I see those students in the follow-up class, for many it’s just the obvious thing to do. Of COURSE you “fix” insignificance.
(I actually have one client at the moment who I keep telling that his results long ago stopped meaning anything but who doesn’t seem to care. We are about to round the corner on maybe 2000 different specifications of his model, but he keeps adding variations and selecting the significant ones. The desire to beat the stock market is a strong one. Plus, hey, more work for me when the result predictably fails to replicate on a second stock exchange and it’s back to the drawing board.)
The second thing that I see is that statistics is seen as a real push-button operation. You pour in the numbers, the smart person pulls the right lever, and the result pops out toasty, warm, and ready to eat.
There’s very little concept that context is important, or that there’s some sort of underlying model. I’ve on occasion gotten shocked surprise when I’ve told clients that I need to know what the variables in the data set actually represent before I build an analysis plan.
It’s also common to just have a data set dropped on my lap so I can go do the statistics to it. It’s very hard to coax a research question or even a preferred dependent variable out of a client who doesn’t understand why you need such a thing. I have the data, don’t I? Just run regressions.
Similarly, the many of them (usually the students or ones with academic training) want robustness checks, but in a very cargo cult way. They see them done elsewhere and they want them too, but they’re not sure why or what purpose they serve. Just add all the robustness checks, please. I need to make my paper better, so add them.
There’s a real desire for statistical analysis to be a completely objective, theory-free, and context-free affair divorced from the reality of where the data came from. And I get where they get that, that’s probably what I’d assume statistics was if I didn’t know more about it.
I should point out that many of my clients fall into none of these holes and grok this stuff well. And the ones that do have these issues aren’t bad or unintelligent. They’re just not statisticians or econometricians (if they were, they wouldn’t need me!)
It’s valuable to know what the folk understanding of statistical analysis is. Even if by “folk” I mean people who have enough stats knowledge to hire a consultant. This is especially important if this is a group of people who are connected, by volume, to a huge chunk of produced analysis.
Whether you’re teaching, consulting, or reading the analysis of others, you want to have a good sense of how people think, and the assumptions and attitudes they’re coming to the table with.
