A Perfect Representation of Myself On Medium in Chart Form
The Dunning-Kruger Effect — Cognitive Bias in Two Graphs
I cannot believe I had not seen this until yesterday and it is from way back in 1999. I think this effect can be even more pronounced when the no-nothing spouting off with great confidence on a certain topic is a bona fide expert in another related field or just an expert in any scientific or technical area. Expertise is generally non-transferable between technical disciplines but don’t try telling that to an expert.
Don’t even think this is going to stop my cross discipline tirades though because it won’t and I will continue to speak with authority on any and all topics when I believe I can back it up. That said my words should be given extra scrutiny when I start talking smack about technical areas outside of microbiology and molecular biology. Nobody gets a free pass but some admission tickets should cost more than others.
I added another graph (Figure 2) from a more recent study which illustrates the same effect. In this instance a group of students were assessed in a variety of ways (without their knowledge) about their knowledge and abilities in a given subject area. They were then told that they were going to be given a test in this particular subject area and each was asked to rate how they thought they would do on the test and also to rate their overall abilities in that area on a scale of 0% to 100% for both. That data is shown along the Y-axis.
Unbeknownst to the students that had already been assigned to quartile groups based on how they had done on these prior assessments. The bottom quartile knowing the least (and demonstrating the least ability) and the top quartile the most. I did not get the sample size for the study but I gather it was decent and certainly statistically defensible. As you can see from the data the perceived abilities and perceived test scores matched up quite nicely across the knowledge/ability quartiles. That certainly makes sense and this just basically says that the students believe the test is fair, it will accurately reflect their actual abilities. (Interesting topic for another discussion, what if the test was intentionally made not fair and biased a certain group and that bias was obvious to the rest of the exam takers? what if the bias was not obvious? Presumably the bias would have to be made clear prior to asking the question about perceived abilities and scores, how could that be done in both obvious and non obvious ways? What if the bias was only revealed during the course of testing, becoming more obvious as the test went further along? What impact would that have on final scores?)
The fascinating thing is how the actual scores correlate to the perceived. Those test takers with the lowest actual knowledge and ability overestimated their abilities by around 45% while those with the most knowledge/abilities actually undersold their performance on average by about 10%. Of course I look at this data and say I am probably consistently underestimating my abilities and knowledge, I would definitely be in that top quartile but I can easily imagine all the people in the lower quartile thinking along similar lines. Thus the more people are given the knowledge of this effect the more pronounced it actually becomes. Now that would be an interesting follow up study. Gotta believe someone has already done it. The field of cognitive bias research is so active I’m sure its been done to death by now.