Are Your KPIs Hurting Your Business?

When you suspect your KPIs are throwing your team off course, it’s time for some serious “ground truthing.”

Teams establish KPIs with the best of intentions. They decide on a definition of success (e.g., increase revenue by 20%) and agree on measurable metrics (e.g., >2% monthly churn rate) to measure progress.

What most fail to consider is the dark side of KPIs. When a team selects the wrong metric to measure progress, it can distort the decision-making process.

Consider the marketing team with a seemingly unattainable sales target for a new product. Naturally, they choose a KPI based around how many new customers sign up. To maximize signups, they set up a 90-day free trial period and automatically register new users for an ongoing subscription. Probably unsurprisingly, this marketing tactic results in a higher degree of churn, call centre volume, and cost of acquisition than had been initially predicted. Even worse, it took several months for the executive team to recognize the disparity between between the signup volume and post-promotion reality. Had they focused on, say, a KPI based around activation and a pattern of regular or increasing use of the service, it could have resulted in a better outcome.

Selecting the correct success metrics becomes even more important when machine learning plays a role.

Kalu Kalu, Co-Founder and CTO of cloud-based photo app Shoebox, can relate. He founded Shoebox as a way for people to relive their best personal moments and share them with friends and family. The more photos you upload and engage with, the smarter the app gets. The result is a highly-personalized experience that keeps users coming back for more.

To ensure their machine learning algorithm was working as expected, his team selected a KPI that initially showed promising quantitative results. All data indicated that the model was learning something meaningful and repeatable. However, when they proceeded to evaluate the algorithm on their own photo collections and a series of independent data sets, they were surprised. The algorithm was, in fact, getting “smarter,” but it wasn’t learning anything that would have helped to increase user engagement on the app. With this qualitative understanding of what the algorithm had learned, the team was able to avoid an A/B test with a potential negative impact on app users and identify a better algorithm to deploy.

Which A/B test do you want your users to experience?

The type of suggestion from an algorithm trained on KPI 1:

The type of suggestion from an algorithm trained on KPI 2:

Another organization we partnered with here at Normative was faced with a similar conundrum and didn’t know it. While the team was aligned internally on KPIs for their new app, they had inadvertently skipped a crucial step: Ensuring that these KPIs reflected interactions their audience actually found valuable. So, we conducted a quick round of in-situ research with their audience to do just that.

Early in the research, we noticed one user had a strong reservation against performing a particular action on the app. So we dug deeper. As we followed this thread of discovery, we unlocked an insight for the app team about their user segments and the content that would be prioritized if they started to optimize performance based on this metric. After seeing a short, compelling video to highlight the problem, the team has now been redirected to consider a completely new behavioural metric — one we think will be a better early indication of a customer’s long-term loyalty — and they’re starting to test it now.

So how do you know if your current KPIs are the most effective ways to measure true progress towards your goals? One exercise we practice here at Normative is called “ground truthing.”

We begin with a simple question: How did you come to decide on your KPIs? Oftentimes, the answers range from “This is the way we’ve always done it” to “This is how everyone else does it.” If there’s one thing I’ve learned in the last decade of helping companies use data to uncover new insights and opportunities, it’s this: History is a bad reason to continue using a KPI that incentivizes bad behavior.

Every organization should make it a point to regularly revisit how they came to decide on their KPIs, and make absolutely sure these metrics are doing their job of keeping the organization on track.

How does your organization think about KPIs? Have a KPI horror story to share? Leave a comment below. I live for this stuff.

Photos © Bill and Peggy Wittman