Why I hate statistics
I have two reasons why I hate statistics. One is rational (it’s not as objective as it is made out to be), and the other is irrational (it’s soulless). Let me explain.
What I do
I’m a researcher. A theoretical linguist. I analyse small data. I rely on the intuitions of native speakers to tell me what they consider a good utterance in a particular context.
So, for example, in Canadian English, people frequently use eh? at the end of a sentence. It’s sometimes considered the equivalent of American huh? And it sure looks like that, as you can see in the two sentences below.
You have the flu, eh?
You have the flu, huh?
Both utterances are used to confirm the speaker’s suspicion.
But through exploring the intuition of native speakers based on a small sample of data, I figured out that eh and huh are not the same. You can see this in the next two utterances.
I have the flu, eh?
I have the flu, huh?
It’s okay to use eh if I want to make sure that my listener is aware of the fact that I have the flu. But huh cannot be used in this way. It sounds like I’m not sure if I have the flu, as if I am asking my listener whether they know. I cannot use this sentence to make them aware of the fact that I have the flu. That’s a big difference in meaning.
This is a robust generalization, as far as I’m concerned. And everyone I have ever told about this contrast confirmed these judgments. I did not need big data.
Now let’s talk about analysis. When I analyse patterns like this, I draw tree diagrams. Because I’m a theoretical syntactician, I care about how words relate to each other. I try to figure out the underlying knowledge that leads to contrasts like those between eh and huh and their possible interpretations. And this can be represented in the form of tree diagrams, like these ones.
These diagrams represent the distribution and meaning of eh and huh. When you ask for confirmation of a fact, you ask about your listener’s knowledge state, and you request a response. What you can do with eh is state that the thought expressed in the sentence (I have the flu) is part of your knowledge and that you wonder whether it is part of your listener’s knowledge state, too. In contrast, it is not possible to use huh to state that the sentence is part of your knowledge.
This analysis (admittedly rather abstract) led me to expect that the same patterns can be found in other languages as well. And indeed, they are. (One of the chapters in my book on the Grammar of interactional language is dedicated to this topic.)
Of course, theoretical analyses rely on certain assumptions. The assumptions I make are well-founded (i.e., they come from published work that underwent academic peer review). But still, the choice of assumptions is somewhat subjective. And many times, the ideas that inspire an analysis are sparked by a stroke of insight. This is subjective creativity. The scientificity lies in the fact that ideas have to be formulated in ways that make predictions that can be tested empirically. They are falsifiable.
So I still consider what I do science, even in the absence of large data or statistical analysis.
What I don’t do
I don’t collect large data. I probably could have crawled the web for all occurrences of eh and huh. I could have systematically worked my way through large corpora. And I could have crunched numbers. But I didn’t. Because I didn’t need to. I have arrived at a generalization that allows me to understand a pattern that occurs again and again across the languages of the world.
Just collecting numbers of occurrences does not tell you anything about how eh and huh are interpreted. Here are the Google n-grams for eh and huh. It shows how they occur over time in the corpus of scanned books available on Google Books.
We can tell that eh has been in use much longer than huh (at least in those books) and that both are on the rise. That’s an interesting fact. But now we have to start figuring out why. And for that, we would have to start looking at the data more carefully to see how they are used. And we would have to annotate the data, which requires some assumptions about what criteria might be useful in the annotation. But, of course, to come up with useful criteria, we have to have an idea of what’s going on. I actualy know what criteria and tags I would use because I figured out the difference based on my small data. They would (at least) involve annotating whether eh and huh are used in a context where the speaker knows if the sentence is true or in a context where the listener knows if the sentence is true. But then, this requires the subjective intuition of the annotator! There will be at least three annotators whose annotations will be cross-validated. But still, we are relying on the intuitions of three people to annotate our large data. So you have not removed human intuition after all.
So, I don’t conduct statistical analysis. I don’t present my results as numbers. Tree diagrams don’t have statistical significance, but neither are they insignificant. The insights they represent could be used to guide data collection for statistical analysis, if you wish to do that.
Why I don’t do (or trust) statistics
I have a visceral reaction to anything bureaucracy. It sucks my soul. It is what happens when I interact with systematized soullessness.
And yes… I have the same reaction to statistics. But its soullessness is only part of my aversion.
What upsets me even more is its pretence of objectivity.
Truth by numbers.
As if!! As if there were no humans behind designing experiments to collect data. As if there were no humans behind annotating the data. As if there were no humans behind crunching the numbers. As if there were no human-made assumptions behind statistical methods.
As if large data revealed more than a single insight.
Statistics is all these things that it is meant to combat: subjective, theory-based, and flawed. But it is camouflaged to the point where human insights are no longer visible. Just to flaunt scientificity.
I rather spend my time (and the time of my students) exploring the equivalents of eh and huh in other languages. And many of these languages do not have large data because they are minority languages. Thus, statistical analysis based on large data is not an option in the first place.
So I keep doing what I’m doing.
Statistics²
One more thing. You can take the apparent statistic objectivity to another recursive dimension and remove the human completely (apparently that is). You can ask ChatGPT to design your experiments and do your statistical analyses. (It’s a thing, yes!) No more human subjectivity involved?
As if again!!
As if the large language model behind ChatGPT was not based on the data produced by humans. As if human biases did not enter these models. As if ChatGPT did not hallucinate and make sh*t up.
As if!!
You really cannot remove the human factor. You can disguise it. But don’t be fooled into thinking that hiding human subjectivity is superior to unconcealedly human perception, intuition, and the insight to which it can lead.