Lies, Damn Lies, and Statistics
We need to stop using statistics to communicate with the public.
Don’t get me wrong, statistics is a valuable tool for certain applications. Informing the public is not one of those applications.
You might be thinking, “That’s a little extreme…why?”
The average (mode) of society is not statistically literate enough to avoid being manipulated.
To combat this, I recommend you read How To Lie With Statistics to get a basic understanding of how people, organizations, and even governments use statistics to further their own agendas.
4 Out Of 5 Companies Think You’re Stupid
At some point in society, statistics became a way to nudge groups of people to do something or believe something.
We’re social creatures and information is limited. Relying on statistics is probably a social heuristic that allows us to make our lives easier by trusting the consensus of experts.
However, that cultural construct has been hijacked for nefarious purposes.
Companies do this all of the time. For example, take the TV ad run by Trident Gum, where they poke fun at the old “9 out of 10 dentists recommend” statement we know all too well.
I admit it’s clever. But how difficult is it for marketers to make a claim based on data to push their objectives?
Shockingly easy. I watched a business seminar by Chet Holmes where he said that there was “data for everything” and that if you wanted to make a marketing claim, there was favorable data for your claim…and for the opposite claim as well.
Chet’s methods were effective, but they didn’t pass my personal integrity filter.
That’s why I recommend we all up our personal stats game. When we see a statistic, the first thing we need to ask is, where is this stat coming from?
Is it coming from a company selling a product?
Maybe an association?
Is the survey conducted by a third party but funded by the company selling the product? The list of questions goes on.
All of this seems overwhelming and a bit conspiratory. But that’s what allows bad statistics to seep through the cracks and into the mainstream: companies know you’re too lazy to dig through the weeds and analyze all the details.
That’s why most of us will take the outcome of a survey or study at face value. Obviously someone had to fact check the results, otherwise, it wouldn’t have been reported…right?
“A river cannot, we are told, rise about its source.”
It’s hard to determine the source of the data used to generate a statistic. Bias can taint the reputability of a stat — when designed in certain ways — can enable people to make claims that are true, but biased.
To detect bias, we need to be aware of the source of the data gathered. One huge clue is the number of people being surveyed or studied. Was it of adequate size and did it have the appropriate diversity, or was it too small or narrow in scope?
For example, if I wanted to find the best steakhouse in Los Angeles, imagine how the results would turn out if I surveyed only vegans? I could easily say I surveyed “locals on their tastes,” but meat eaters would think I was crazy.
Data without context (knowing assumptions and biases) can lead to incorrect conclusions.
Which Average Do You Mean?
What’s your average weight over the last 30 days? Before you call me rude, I’ll tell you mine.
165.1, 167.4, and 163.2 US pounds.
How can I have three average weights? Well, first let me ask you, what’s an average?
Well, it could be either mean, median, or mode…
This slight nuance enables statisticians to lead us to different conclusions.
Let’s stick with my weight example:
- Mean: I added up my 30 days of weights and divided by 30.
- Median: I listed my weights in ascending order and picked the absolute middle value.
- Mode: Which weight over the 30 days showed up the most times.
Isn’t it crazy that we can be led astray by such a simple concept as an average?
If you’ve never heard of Death Valley, CA and saw the average temperatures and the image below, how would you pack your suitcase for a vacation there?
Sounds like a perfect getaway destination and also includes a complimentary rainbow for a tropical flair…
Oh, it’s also the hottest place on Earth! Temperatures have been known to get over 130 degrees Fahrenheit. And if you went during the summer, you’d be in for a rude awakening.
“An unqualified average is virtually meaningless.”
I hope this illustrates just how deceiving averages can be if not properly understood.
Don’t Confuse Normal With Desirable
I’m getting to that age where everyone around me is getting married and having kids. As good parents, they all want what’s best for their child, so they consume books in the hopes that they’ll give their kids a head start.
As they read these books, their brains are primed to look out for certain milestones their child needs to hit so they know their child is on the right track.
But what happens when a parent starts to freak out because their child didn’t start talking by six months of age? Heaven forbids the kid decides to speak up at month seven. Just think about how that parent feels the longer their kid isn’t progressing “normally.”
When we read statistics, it’s important that we don’t overlay our perception of normality on the natural process of life. Recently, I had lunch with Kevin who was joking that his neighbor brother didn’t speak a word until the age of 2, but now he makes up for it 10-fold.
Generalized statistics can’t factor in for all variables, so don’t rely on them to always fit your personalized situation.
Don’t Be A Sucker
There are countless more examples of how statistics can lead us astray, so we need to adopt a new lens to separate the wheat from the chaff.
Here are a few rules you should keep in mind:
- Be a skeptic. Don’t take a stat at face value until you look deeper into who’s telling it to you and what their motivation is to get you to believe it.
- Look for bias. This is hard because you’re not the one doing the survey or study, therefore you’ll need to get your hands dirty and look at how the data was gathered. If this is not possible, you’ll need to rely on the reputation of the entity stating the results, so make sure they’re legit.
- Ask, how do they know? If you can’t look for the bias yourself, then you need to put the organization stating the statistic under a microscope because you’re relying on their reputation to say correct findings.
- Are there comparable findings by other organizations? If there isn’t consensus between multiple third parties, then there is probably bias you need to be aware of. This can be difficult because one organization might have a proprietary method that is better than its competitors.
- Is there any error margin listed? In measuring things there is usually some margin of error, either by the tool or by the conducting human testers. If you don’t see any error listed, it’s probably not credible.
- Be aware of averages. I bet you’ll never look at them the same based on the earlier examples.
- Humans can interpret data in different ways. We all have different life experiences and beliefs, therefore if two people look at the same raw data their conclusions can still be different. The hard part about studies paid for by companies is that it can be hard for the study executors to “bite the hand that feeds” by reporting unfavorable results.
- Ask, “Does this make sense?” After you’ve run it through all of these mental checks, do a gut check. Be extra scrupulous of generalized future trends based on past data as it can be difficult to predict accurately without using proper methods.
I want you to find a statistic you believe is designed to mislead you.
If you find one, shoot me a tweet at @JonStenstrom with your finding and why you think it’s bad.
I’ll leave you with a quote from Mark Twain:
“There is something fascinating about science. One gets such wholesale returns of conjecture out of such a trifling investment of fact.”
Be diligent in finding facts and only form conclusions based on credible sources.
We need to stop using statistics to communicate with the public. Don't get me wrong, statistics is a valuable tool for…thoughtstack.libsyn.com