Can’t see the goal for the metrics
There’s no doubt metrics and KPIs are an essential part of building a successful internet economy business. When chosen wisely they can help guide you towards your strategic objectives. They enable you to measure the impact you’re having, draw conclusions from experiments, and gain insights into people’s behaviours.
Metrics are vital but they are also seductive. It’s easier to say we need to improve a certain rate than to communicate a complicated goal. Easier to make an important metric the objective than to define a vision. Easy to absolve people from understanding the complex nuances of human desires, emotions, frustrations and habits.
Regression to the metric
It’s tempting to justify it by saying the metric is just another way of expressing the underlying value for the customer so they are actually at the heart of it. Even with the best of intentions though, people become detached from the original reasons and over time the purpose behind the metric slips away.
This can lead to people unwittingly gaming the system. In a recent experiment a team whose goal was to increase activation rate tested a page containing 10 flight destination deals. They removed the descriptions of each destination in an experiment and discovered activation rate increased. A different team had the goal of “helping travellers find deals to destinations they’d be interested in visiting”. They ran a series of usability testing sessions and learned that people need more information to be able to properly compare the destinations and decide whether to visit them. As one person put it “it’s not like buying a packet of cereal”. The page with the descriptions provided more long term value despite having a lower activation rate.
“It’s not like buying a packet of cereal”
Sometimes the consequences of unintentionally gaming the metric can be more tragic. Setting the goal of reducing hospital Accident and Emergency waiting times to below four hours seemed to be in the best interest of patients. Unfortunately it led to nurses and doctors prioritising minor cases over patients with life-threatening injuries, simply because they had arrived first and had been waiting longer.
It’s tempting to think “I wouldn’t let that happen” but if it can happen when lives are at stake then it can happen when app conversions are at stake.
Avoid pitfalls, solve problems
If you manage to avoid this, having a metric as your raison d’etre can still leave you in an awkward position. It becomes very hard to justify doing something that lowers the metric. Even if that benefits the user and is in the long term interest of your business. It can lead to teams not tackling issues or not pursuing opportunities because those would drive a metric that was another team’s responsibility.
When the metric is put on a pedestal, when it becomes the purpose of your team, you risk missing the big picture. If the metric is the goal it doesn’t leave you any room to think about whether this is the best way to solve the problem or whether that metric is still taking you in the right direction. It’s not flexible enough to enable you to see other options and objectively assess your position.
A metric is not a vision and it’s hard to build a strategy around a metric. Instead focus on what matters: the people that you want to use your product and how you can make their lives better. Or in the words of Kathy Sierra, how you can make them badass. What are they trying to achieve? What problems are they facing? Discover their jobs to be done. Then prioritise and organise around these, choosing the most appropriate metric to measure your progress towards addressing each job.
“A metric is not a vision.”
The right tool for the right job
Metrics are tools. A compass will help you go south in search of warmer climes but it won’t tell you about the oceans you’ll need to cross and at some point you’ll end up in Antarctica. A map can show you the possible route options and together with a compass will help you navigate to an appropriate destination.
Metrics help us make decisions and keep us honest but they are not the goal and they are not a substitute for thinking.
Other popular posts:
- “A nightmare, a nightmare, and once again a nightmare” — but not a failure
- Does falling off a cliff make you a better employee? Reflections on rock climbing and product management.
- 8 Things I’ve Learned Product Managing Car Hire during the journey to 10 times growth within 2 years!