The KPIs of KPIs

I am running a bootstrap startup. The code is in good shape. The testing coverage is so good that I can deploy at any time without QA cycle. The product is stable, eventhough the code is extremely complex.

I have all of that, without settting KPIs for number of bugs or code coverage.

The story of the KPIs (key-performance indicators) in software corporates is funny and painful.

Have you ever felt you are enforced to meet some numbers, that everyone knows it has nothing to do with the success of the company? Are you required to work on KPIs, just because they are marked in red over a dashboard at the middle of the office? Are you familiar with some weird blitzes at the end of the quarter / year, for fixing numbers?

In corporates, managers see these numbers as a good recipe for a successful company. I am not so sure they right.

Here are two examples for KPIs I had to meet during my career:

Opened / closed bugs ratio: I’m still waiting for the executive manager to say “amazing! What a great impact did you bring by reducing this annoying ratio”.

Static code analysis warnings: I’m always happy to come back home, and tell my wife “you have no idea how important my role is! We now have only 132 minor warnings in our code analysis tool”.

Let’s list some rule of thumbs to know whether your KPI will ever contribute to the overall performance of the company:

1. You are embarrassed to talk about the KPIs with senior management: It will be a bit weird to show the CEO your KPI dashboard, filled with “number of P1 bugs” per development group.
2. Amorphic bazzwords: Are you willing to pay for someone that promise you to deliver “quality”?
3. Task is missing from the backlog: Have you ever sat in a planning meeting, and the manager / product manager said — “hey, but if we will implement all of that we won’t have time for the KPIs”? KPI needs planning, effort, trial and error. It shouldn’t be just an annoying task upon angry manager email.

What is a KPI?

KPI is the way to measure the results of the actions you take from a goal defined by higher management level (I’m not sure it’s the definition in the dictionary, but the dictionary never run a company, and certainly didn’t set KPIs...).

Your manager set you a goal. In order to meet this goal, you need to break it down into actions. Then, you need to set metrics that will measure whether you succeed to meet the goal.

For example, Chris, the CEO, set a goal of reducing customers churn rate. Obviously, reducing the churn rate will increase company growth. What are the action items required to gain that? After talking to customers and account managers, it turns out that the product is not intuitive enough, and the response time for the customer’s inquiries takes too long. These insights match the NPS survey made last year, that shows how customers are disappointed.

So the CEO set the following KPIs table:

Great! We know what the strategy is, what needs to be done, and how we measure ourselves! Not only that, Now the R&D knows what its goals are: Improve the product UX and give quicker respond to customers. But how can they achieve them? What actions the R&D needs to take in order to say “we improved the product UX in a way that will cause the churn rate to reduce”?

The product and the UX team sat on this problem and discovered, with massive usage of Google Analytics, that it takes around 3–5 minutes to complete the setup, and only 3 main features are being used, out of 12 in the “processor configuration” screen. Also, they found that the loading time of the main page is around 30 seconds, which makes users to give up.

So the R&D managers, set the following KPIs table:

When John, a team leader in the R&D stand up and said — “ ‘ Shorten the setup time up to 3 minutes‘ is the measurement, not the action”, his manager answered — “Nope. ‘Setup time less than 3 minutes’ is the definition of done for the action you take. It says nothing about achieving the goals you set”.

See what we are doing here?

We are not setting KPIs just because. We are starting from strategy, and we go down the pyramid and verify the strategy is being implemented. Furthermore, in every level we know how to measure a success.

Let’s go back to the “Opened / closed bugs ratio” metric mentioned above. Isn’t it a good metric? Can’t we put it on the table in the same way?

Well, let me ask you this: Once you meet this KPI number — does it mean the product was improved? If I’ll ask the customer — will he say “WOW, you are much better now”? In other words: Do you measure the action you take or measure the goal you set?

Wait a minute. Maybe the “make product more stable and less buggy” should be a goal for a team within the R&D?

This is much better. The company goal is to reduce churn; The way to do that is by improving product UX, which become the goal for the R&D; The way to do that, is by making the product more stable and less buggy, which become a goal for a specific team. The way to make the product more stable is by closing the bugs — which can be measured by opened /closed bugs ratio.

Bingo? not so fast.

Indeed, this is a logical thinking. In some cases, this is how KPI structure should look like. Nevertheless, please consider the followings:

1. Do you have a reason to believe that the problem is the bugs? Did you check that the bugs that were found in lab and were not solved are coming again from the field? Maybe there is lack of testing? In other words: Does the ratio measures the success of “closing bugs” or “product less buggy”?
2. Do you believe that the customers will see the impact of your action?
3. And most importantly — did the KPI represent the way you think about breaking goals into actions, or it is just a metric that you saw over the internet, and you’re trying to justify it by building this table bottom-up?

The problem with how managers set KPIs is that they confuse between goals and means. They are mixing between measuring a success and how to drive people to work on something.

During the process of building KPIs as described above, we gain these benefits:

1. We distinguish between goals and means: Reducing churn rate is a goal. Reducing number of bugs is a mean. Measuring a success is measuring whether we are meeting the goals we set. Not the actions we took do implement that.
2. You need to plan KPIs: Achieving the goal of reducing churn is a huge, tough task. It needs a plan: Which things impact churn? What are the main customer’s concerns? How much effort do we need to put in order to reduce this number? “Quick respond for customer’s inquires” should be a result of research and analysis. This research is super important and, unfortunately, less considered.
3. Working on essence, not on number: It’s not just a task at the end of the quarter, it’s the way of thinking. It’s the way we build the company. Every developer knows that he needs to respond fast for escalations, because that’s what bothers the customers. This will reduce the churn — which will make the company to become successful.
4. Aligning the company with the strategy: It doesn’t matter whether you are executive manager, or a developer at the bottom of the pyramid. You know what you are doing, why are you doing it, what are the impacts in company level, and what will be considered as a success.

Back to the rule of thumbs for bad KPIs.

Let’s examine this table again:

Can you show this table during the all-hands session? Is it something that is interesting to show during the monthly status with the CEO?

Is it amorphic? Are you willing to pay for outsourcing service, that will ensure the next NPS survey will produce results that are better in 20% than previous survey?

Is it a real task that needs design, product and thinking? Do you need a story in your project management tool that describes “shorten the setup time up to 3 minutes”?

Summary

KPIs are not just numbers out of the blue; They are a way of thinking, the way you break goals down into tasks and measure how much you reach the goals.

KPIs are hierarchical. Every management level set goals to its direct reports. Together they break the goals into tasks. Every task become a goal to the next managerial level, down stream.

After understanding the model, it’s easy to see how KPIs can drive the company forward. How it makes everyone aligned to the strategy, how it is effectively measures the performance.

Back to my startup.

My strategy was to keep the churn on 0. No customer should leave. It takes too much time to secure new customers, I cannot afford myself to let someone leave. In addition, I wanted the time-to-market to be as fast as possible, so I can move fast with new leads. Luckily for me, I managed to be where I planned. How did I do that?

• I have great testing coverage
• Every field bug is fixed the day after, with a new automated test that covers it.
• I have full CI/CD. I can deploy my code at any time.

I never set quality as a goal. It wasn’t interesting enough.