You Don’t Know What You’re Measuring
Most studies make a claim. Students who think about growing their skills do better. And so on. However, most of these studies have a dirty little secret. Their measurement tools don’t measure what they study.
We can’t measure concepts (e.g., constructs) like creativity, intelligence, happiness with a machine or a ruler. Most measures for happiness come from 7-point scales. I don’t know about you, but I’m feeling pretty 4 today.
Asking people to report, reliability how they feel about something is problematic. I have a colleague who studies taste. All day long he asks his participants to describe how good something tastes to them on a 100 point scale. To try to establish a common baseline, he asks each participant to describe the worst experience in their life, usually a death in the family (that’s the 0 on their scale). He also asks each participant to describe the best experience in their life, usually sex (that’s the 100 on their scale). So where would eating cake lie on the scale from grandma’s death to an toe-curling orgasm? Most of the time, he says, way closer to an orgasm then you would expect. Really? Is this graduate-student produced cake a 98? Asking for people to self-report feelings is not an accurate measure.
Ideas, concepts, feelings, and other distinctly human things don’t follow the rules of nature. We can and should not apply the same habits of science. We should not treat data about feelings and ideas the same way we treat data from the natural world. When you read a study about human feelings or concepts, do me a favor, check to see how they measured those constructs. Take those measurements with a grain of salt.