Using Data to Augment Decision Making
What you desperately needed to know about KPI's but never realized you should ask.
When it comes to business decision-making tools, KPI’s are many people’s favorites. They are easy to understand, generally easy to measure, fairly straightforward to interpret in relation to the desired outcome, quick to obtain… and yet they are not particularly well-known for helping improve the organizational performance they are supposed to correlate reliably with.
Why? Because KPI's leave the decision modeling to humans. And as mere humans, we are really bad at changing our minds. This is called confirmation bias. We are champions at making up our minds, and then selectively interpreting information we are presented with in a way that confirms we've made the right choice. If we bump into disconfirming evidence, we tend to look away. And if someone points it out, we dismiss it.
There are many anecdotes of KPI's being used flexibly across organizations to defend decisions or assessments which are, upon closer scrutiny, only loosely supported by data. "I don't have a performance problem in my team, we achieved 71% of our OKR's this quarter". Someone else looking at that same data point could draw an entirely different conclusion depending on whether they're a glass 71% full or glass 29% empty kind of person. Who's right? Either, or both, or neither.
Wasn't there a quick fix for this? According to decision scientists, if you're really enamored with KPI's, you should agree on a threshold for deciding one way or another before peeking at the data. The trouble with agreed-upon decision-making thresholds is that, eventually, someone is bound to question the agreement. If you want to laugh, you should look into the language used by psychological scientists to refer to p-values larger than or equal to .05, the agreed-upon threshold for calling an effect statistically significant. Wait, aren't they in the field that discovered confirmation bias in the first place? Exactly! Confirmation bias is that strong.
The more robust fix involves changing the decision-making framework altogether, to something both more precise and less prone to confirmation bias. Start with a default choice, and systematically weigh the existing data against that default choice. If the data is better aligned with the alternative, go with the alternative. If this reminds you of statistics and hypothesis testing, you got the gist of it. In fact, the most compelling definition of statistics I came across belongs to a decision scientist I have a nerdy crush on, Google's Cassie Kozyrkov.
"Statistics is the science of changing your mind under uncertainty."
Together with its not-so-distant cousin Machine Learning, the field of Statistics is a game changer for decision-making in organizations because it provides systematic methods to empirically derive the decision itself, optimized relative to a desired outcome, for example increasing revenue or decreasing costs.
Some of you will astutely point out that KPI's are also empirically derived. They should be. However, many of them are nothing more than a rule of thumb derived from a different population, under a different context. As such, even when they are used correctly, they are prone to error. The more pernicious issue is, even for those KPI's that are applicable and precise enough, by design the actual weighing of the different choices is left to the human in charge. This leaves plenty of room for human bias to be introduced. And better KPI's won't get you out of this conundrum.
What kinds of decisions should you use Statistics or Machine Learning for? The kind where the stakes are high (someone has a lot to lose or a lot to gain), the kind that are in fact numerous decisions that together impact on the bottom line (which of your 3.2 million customers are more likely to churn than not), and the kind that human decision-makers tend to get wrong (hint: most don’t keep a track record).
If you want an analogy, the premise of the movie 21 (2008) provides a pretty good one. The difference between a decision making framework based on KPI's and one based on statistics and probability is roughly equivalent to that between blackjack strategies based on heuristics (quit while you're ahead, avoid basing your outcome on previous bets) and counting cards. If you can pull it off, counting cards will make you more money.
