Dunning-Kruger Confound-Spotting
Or: Why you aren’t smarter than a professor. Probably.
One of my writing idols is Tim Urban of Wait But Why. He’s excellent at creating accessible, aesthetically appealing art that represents important concepts — concepts like the Dunning-Kruger effect:
This phenomenon is based on the work of two researchers, Dunning and Kruger, that showed less competent people tend to overestimate their abilities, while the inverse is true of competent people. While Tim Urban’s graph makes intuitive sense, it’s not quite a fair representation of the two researchers’ findings (Tim admits that, too); The graph of confidence versus real competency looks more like this:
So it’s not quite the case that the incompetent are more confident than experts, only that they drastically overestimate their ability. People simply tend to think they’re above average.
Much ink has been spent investigating where we see Dunning Kruger effects in real life. In particular, I recall a period where every other news article on my feed claimed that Donald Trump is the “Dunning-Kruger President”. But this isn’t limited to the president — we all fall victim to overconfidence in our abilities at one point or another.
One area where I think people consistently overestimate their ability is in the evaluation of scientific research — in particular their critiques of research methodology.
We begin, of course, with Twitter:
This was inevitably picked up by Tumblr users as another example of how research doesn’t acknowledge class distinctions in horse ownership.
Here’s my objection: do you think the researchers didn’t think about that? It’s trivial to control for income, employment status, and other factors that may be confounds. The people gleefully retweeting, liking, and commenting on this post must think they are more competent than the original researchers. In other words, they’re at the top of Tim Urban’s Dunning-Kruger mountain — the area where you think you’ve got it all figured out.
You can see this in how people respond to the COVID case numbers. “Oh, the numbers are higher because we’re testing more.” Again, this can be controlled for — metrics like test-positivity rate exist, among others — so this objection often falls flat.
You can see this in how people respond to contentious issues like racial bias in police forces. I can’t tell you how many people claim that the white/black disparity dissipates when you control for relative crime rates and poverty. Surprise! Even when controlling for those factors, there are still large discrepancies.
Sometimes it’s more complex. Gender wage gap skeptics often claim that controlling for job preference and years of seniority or education makes gender disparities dissapear. They mistake comparisons of median wages of men and women to claims about equal pay for the exact same job. Gender wage gaps still exist when controlling for career, seniority, and education — there’s a roughly 5 cent difference in pay between men and women that is left unexplained.
Dunning-Kruger or Confirmation Bias?
I’ll admit there’s some crossover between Dunning-Kruger effects and confirmation bias. Too often the studies people stridently accuse of poor methodology are in some way linked to their personal politics. Accusing a study of not adequately controlling for confounds you just thought of is then a great way of preserving your prior beliefs.
I’d offer that the two biases seem structurally similar. We want to believe we’re good at stuff, and look for confirmation. When evaluating our own skill level, we ask ourselves “can I believe I’m good at this?”, and not “must I believe I’m good at this?”.
Either way, here’s my secret advice to mitigating the problem I’ve articulated: read the study. You don’t need to be an expert scientist to read a paper and check if the authors controlled for whatever confounds you think would be important. If you think the horse ownership study doesn’t account for health insurance status, why not look? You have nothing to lose but your own overconfidence.
