On Metrics and KPIs
Metrics, also referred to as Key Performance Indicators (KPIs) in more limited contexts, are a staple of the modern business world.
It’s considered common wisdom and “best practice” that “you can’t manage what you can’t measure”, and therefore that you should measure it. It is also fairly well accepted that the measurement in an of itself contains little insight, and that the insight is generated from comparing the actual measurement, to a forecasted target that was set ahead of time. This is roughly where common wisdom stops, leaving out several critical aspects of using metrics effectively. I’d like to cover two of these aspects today.
1. Targets must be relative and dynamic
[I]t is impossible to set a target in advance that represents ‘good performance’. The modern world is complex and dynamic making prediction impossible, and we know that we will never face the same set of circumstances twice. The economy changes, as do our customers and competitors, so we can do no more than hazard a guess at ‘what good looks like’
[A]ny measure of reality will always contain noise: the impact of an unknowable number of random or irrelevant events, which distort and disguise ‘real’ performance (the signal)… [I]f we do not have the ability to measure the level of noise, we have no scientific basis to distinguish between something that is safe to ignore and something that we should be acting upon.
…so most of what passes as performance analysis is the result of comparing a guess with noise!
Worry not. This does not mean that any metric and any measurement is useless. The folks at BB also offer a solution:
The only way that we can assess performance in a truly meaningful way is to compare ourselves to peers that have faced the same set of conditions.
Targets, should therefore be relative: at the business level — compared to external competition; Internally — compared to other teams/departments if feasible. Here’s a good example from StatOil, defining its shareholder-return target to be “above peer average” and its return-on-capital target to be “in the top-3 of peer group”.
2. Metrics should be paired
“If you give a manager a numerical target, he’ll make it even if he has to destroy the company in the process.” -W. Edwards Deming
Leaving the “gaming the system” challenge for a different post, on a less malicious level, a key challenge with metrics is their reductionist nature. Over-orienting behavior towards a view that looks at the world through a single number is unlikely to lead to a positive outcome. Here as well, “not measuring” is not the only solution. And you don’t have to go full-on “balanced scorecard” either. Keith Rabois offers a simple idea that’s more lightweight to implement:
One important concept is pairing indicators. Which is, if you measure one thing and only one thing, the company tends to optimize to that. And often at the expense of something that is important… It’s really easy to give the risk team the objective and say, we want to lower our fraud rate. It sounds great. Until they start treating every user in this audience as a suspect because they want to lower the fraud rate. So they require each of you to call them up on the phone and give them more supplemental information and fax in things. Then you have the lowest fraud rate in the world, you also have the lowest level of customer satisfaction score.
What you want to measure at the same rate as your fraud rate, is your false positive rate. That forces the team to actually innovate. Similarly, you can give recruiters [volume] metrics around hiring. And guess what? You will have a lot of people come in for interviews. But if you are not tracking the quality of interviewers, you may be very unhappy about the quality of people you are hiring and giving interviews to. So you always want to create the opposite and measure both. And the people responsible for that team need to be measured on both.