The fallacy of data-driven decisions

An essay (and somewhat of a rant) about how to make good decisions.

“Show me the data.” “What evidence do you have?” “Prove it.”

If you’re like me, these words probably evoke chilling memories of times when you went out on a limb to propose an idea, but rather than being met with intrigue and enthusiasm, you received a punchy objection. So you back-pedal, stammer out a few words about not really having evidence, and then retreat into a corner of sheepishness swearing to yourself not to make that mistake again.

What is data anyway?

Data (noun): facts and statistics collected together for reference or analysis.

Data has become the number one challenge facing the world today. In an age of nearly unlimited access to information, we are struggling to make sense of everything. At any given moment, we are a few clicks or taps away from the answer to just about any question we have.

We have such instantaneous access to answers that we have become addicted to it. We search Google anytime we have a question. We check peoples Facebook profiles rather than asking them whether they have a girlfriend. We browse Quora for fun.

But isn’t this a good thing? Shouldn’t we be celebrating people’s curiosity and thirst for knowledge? Aren’t Google and Wikipedia helping to advance public education?

Data is flat

The problem with data is that it’s flat. Data by itself has no inherent meaning, it doesn’t tell a story, it doesn’t compel you. Of course, many people have used data as part of a good story or compelling argument, but the argument is not compelling because it has good data, the data is compelling because the argument is good. Read that again:

The argument is not compelling because it has good data, the data is compelling because the argument is good.

Many an argument has been made using good data, but the argument is not good because it does not lead to any new insights or helpful information. You may have heard the alarming statistic: When ice cream sales increase, shark attacks increase as well. The assumption is that ice cream causes shark attacks, but the astute among you will notice that this is a case of the common-cause fallacy. This is not a good argument because these two things are in fact caused by a separate thing which is warm weather.

Alternatively, many good arguments have been made without using data. Consider for example the famous I Have A Dream speech by Martin Luther King, Jr. One of the most famous speeches ever — and certainly one of the most compelling human rights arguments in history—was not based on hard facts or scientific research. Mr. King performed no controlled studies nor did he present statistical evidence of his position. No, he told a story. He told a story so powerful that it shaped a generation of politics and society, one that continues to be remembered and inspirational today.


You’re missing the point

I’m sure many of you are thinking I am totally missing the point of being data-driven. There are many examples of terrible decisions which were made because someone failed to look at the data. You’re probably thinking:

“We need people to think critically about their decisions and use real hard facts to inform them rather than bogus assumptions.”

I hear you. You’re saying that decisions made without data are just guesses, or worse yet, shots in the dark. Surely data can help us make better decisions than taking random chances and hoping it works out.

Well you’re right, but the problem is that data is just one part of the equation.


What ever happened to intuition?

When did intuition go out of style? When did wisdom and experience become passé? Since when is not ok for someone to “just know” something based on their various skills, knowledge, and life experiences?

When did we stop trusting people to know what they’re talking about and start believing that they’re wrong until proven otherwise?

One of the greatest aspects of humankind is our ability to collect, retain, and share wisdom and common sense in our lives. Without this we would be constantly overwhelmed by the amount of decisions we would have to make and re-make everyday. Instead, our brains retain lessons learned so we can easily (read: instinctively) react to similar situations it the future. As these “insticts” enumerate over time, we call them wisdom, and we uphold wisdom as something to be respected and strived for.

The “data-driven movement” as I’ll call it moves us in the other direction. The goal shifts from increasing in wisdom so you don’t have to make more decisions to increasing in scrutiny so that every decision requires more data to justify it. We are moving from smart intuition and fail-fast thinking to analysis-paralysis.

The means justify the ends

Let’s be honest, humans are biased. We are unequivocally subject to opinions and our own interpretations of the world, whether we have data or not. By and large, we make decisions based on what “feels right” and we try to back it up with reason. What happens so often with “data-driven decisions” is that we form an opinion and look for data to back it up.

Data is only as good as the data you have. You don’t know what you don’t know. So you find data that says you should do X. What about the data you didn’t look for that actually proves you should do Y? How do you know you’ve collected all the data? The right data?

Data isonly as good as your ability to interpret the data in a meaningful way. You can get all the data you want, but you can’t act on it if you have no idea what it means. How do you come up with meaning without drawing on your own or a group’s collective wisdom and intuition? This inherently introduces some bias into the interpretation and the resulting decision.

You see, rational minds are easily convinced with arguments that sound rational. Whilst emotional minds are easily convinced with arguments that evoke emotion. Taken alone, both perspectives are biased. True objectivity requires a balance of reason and emotion, data and intuition.

Decisions are risky, period

What you mean when you say you want data to inform a decision is that you are uncomfortable with the risk of uncertainty. But decision-making is inherently risky. There are always alternatives and unknown outcomes. There are always risks. Sometimes the best way to resolve risk is to take a chance and learn from it. You can waste your life in analysis paralysis and never get anywhere, or you can follow your gut, measure the results, and learn what works.


A call for less data and more decisions

We live in a world with overwhelming amounts of data. We have entire careers, businesses, and industries dedicated to collecting, storing, and making sense of our data. We spend hours/days/weeks pouring over data trying to make sense of it so we can make a decision. We nit-pick every little thing to try and ensure it is “right” and “justified” and will “work”.

What if we analyzed less data and experimented more? What if learned from our successes and failures, gained individual and collective wisdom, and could make the right decisions with more consistency and in less time? What if we trusted people with experience to know what they’re talking about and empowered leaders to do the right thing as best as they know how? What if we focused on the decisions with the highest potential reward when made right—the impact?

What if we made more impact-driven decisions?

Epilogue: I hope you enjoyed reading my thoughts on data-driven decision making. By no means do I claim to have all the answers. My goal is simply to explore the nature and challenges of decision making and present a counter-cultural perspective. I welcome your thoughts, opinions, and disagreement!