Stop Measuring The Pizzas And The Cooks

Why output-oriented metrics are mostly a waste of time in software- and product development

Want to listen instead of reading? You can also listen to us reading this post in this episode of our podcast.

Good news! You’ve become the owner of a pizzeria. Gone are the days of writing code, being a Scrum Master or deciding over the Product Backlog. Instead, you are now responsible for a small Italian-themed restaurant. From now on, your world revolves around anchovies, mozzarella, tomatoes, and basil. But there’s one thing you took with from your job as a Scrum Master; use empiricism to drive decisions. Instead of relying on assumptions, intuitions, and expectations, rely on data to improve how you do your work. So what are metrics you should track for your little shop?

“Instead of relying on assumptions, intuitions, and expectations, rely on data to improve how you do your work.”

Metrics for pizzerias

On face value, a couple of metrics might suggest themselves. You could measure the number of pizzas produced on a night. More is better, right? Another option is to measure how many people are in your kitchen and how much time they spend working there. Again, more would seem better. And how about the number of people walking into your shop? The more you have, the better you are doing.

Except …

None of these metrics are helpful as they are only concerned with output. They measure only how much work is done. They don’t tell you anything about the value of that work; the outcome. You could do really well on all of these metrics and still run your shop into the ground. You can create hundreds of pizzas every night. But without customers to buy them, you have only created a heap of waste and lost money producing it. The same happens when you have dozens of people working in the kitchen, putting in many hours of overtime. Without actual customers, you’re losing a lot of money and generating waste. Even when your shop is succeeding in attracting hundreds of people, your shop will still go belly up if they don’t buy anything.

No entrepreneur in their right mind would measure these kinds of metrics for a pizzeria. When you focus on output-oriented metrics, you are likely to optimize for how much work is done. How can you optimize the process to bake more pizzas? Let's buy faster ovens! Let’s send our cooks on expensive training! Let's hire more people! Let’s advertise more to draw in larger crowds. Instead of optimizing the process, output-oriented metrics tend to encourage the accumulation of (invisible) waste in your process.

“Output-oriented metrics tend to encourage the accumulation of waste.”

So, measure outcomes instead

If you want to reduce waste, a much better approach is to measure the outcomes of the work. How many pizzas are you actually selling? What is the margin of profit on each pizza? How happy are your customers with their pizzas? How many people return for more? What percentage of the people walking in actually sits down to order? How rapidly are customers being served? What is the ratio between how much you spend and make for each week, month, quarter or year? How many pizzas and ingredients do you throw away at the end of a night?

What these metrics share is that they don’t measure the work itself, but the results of the work. And measuring them allows you to pinpoint where waste is happening. And they give you more meaningful information about where improvement is actually necessary. If customers are generally unhappy with their pizzas, you need to improve the quality or diversity of the menu. If the margin of profit for a pizza is too low, you need to either increase the price or find ways to lower the price of production. If you’re not selling enough pizzas to cover the costs, you have to rethink your model. Perhaps there are already a dozen pizzerias nearby, and you are better off selling other kinds of food. If the number of visitors that convert into buying customers is too low, you may have to rethink your menu or the pricing.

It can be harder to measure outcomes …

Now, outcomes are often harder to measure than output. Measuring outcomes requires a good understanding of what your value is and to what degree that value is reaching your customers (in itself a good sign that these are the metrics that matter). Given the difficulty, a concession might be that you’ll just have to do with output-oriented metrics because that’s what you’ve got. An underlying assumption here is that output and outcome are related, meaning that if you can at least measure output, you’ll have a good sense of what your outcomes are even when you don’t measure them.

But it's important to realize that output and outcome are mostly orthogonal— one doesn’t cause or result in the other. And it's easy to understand why. You can do really well on your output (many pizzas produced by many cooks) and still go bankrupt (low sales, unhappy customers). You can run a profitable business (high margin of profit, many returning customers) even when your output is low (few pizzas produced, small staff). Makes sense, right?

Back in the world of software development …

So if it makes sense, why is it that we’re obsessed with output-oriented metrics when it comes to software development? Measuring how many items or story points are completed per Sprint is the same as measuring the number of pizzas produced. It just doesn’t help. Tracking how many people or teams are working on a product and how many hours they are putting in, is just as effective as deciding how successful your pizzeria is by counting the number of cooks in your kitchen.

“Measuring how many items or story points are completed per Sprint is the same as measuring the number of pizzas produced. It just doesn’t help.”

Coincidentally, it seems to me that the strong focus on output-oriented metrics is a huge driver of scaling efforts in many organizations. Without considering how much value is actually being generated, adding more people and teams seems like the way to go to generate more value.

So if not output, what are useful outcomes to measure in this context? When you work with the Scrum Framework, or any other Agile approach, the focus shifts from working as hard as you can to delivering small and valuable increments of a product to stakeholders so you can learn with them what else is useful. So we would expect outcomes in four areas; increased responsiveness to changing needs, more valuable outcomes and improved quality. Overall, we would also expect improvements in each of these areas over time as people learn how to collaborate more effectively as they iterate together.

Below are some of the metrics that capture aspects of these outcomes. None of them are perfect. None of them should be measured in isolation. And all of them should be inspected by everyone involved in order to make sense of what they mean. But if you want to measure anything in your process, these are a good start:

Metrics for Responsiveness

• Lead Time or Cycle Time: how much time transpires between when a need is identified (lead time) or when a team starts work on it (cycle time) and when it is released to stakeholders?
• Work-in-Progress: how much work is ‘in progress’ at the same time on different levels of the system?;
• The number of interruptions: how many times are Scrum Teams interrupted from their focus during a Sprint?

Metrics for Quality

• Code Quality: the quality of the codebase with static code analysis, test coverage or whatever makes the most sense;
• Total defects: the number of known defects in your software;
• Stakeholders happiness: how happy the stakeholders are with the quality of what is being delivered to them;

Metrics for Improvement

• Team Morale: how willing people in your team and its stakeholders are to collaborate together, even when the going gets tough;
• Innovation rate: the percentage of time spent on new technologies, skills, and approaches;
• Number of dependencies: the number of external dependencies that teams have to deal with in order to get work done;

Metrics for Value

• Return on investment: the financial value of the work that is done by the team versus the cost of creating that value;
• Conversion rate: the number of visitors and people that turn into paying customers (not always relevant);
• Profit margin or revenue generated: the profit margin on work done or the revenue generated by it (depending on the kind of company);

You can find more about the kinds of metrics to use by exploring Scrum.org’s Evidence-Based Management framework.

Closing words

Stop measuring the proverbial pizzas and cooks in your work. Tracking metrics like that only tells you about how much work you’re doing, but not how valuable that work is. For all you know, you’re creating hundreds of pizzas with dozens of cooks that nobody wants; its a complete waste. And that is the underlying problem of output-oriented metrics. They encourage the creation of waste in your process if you don’t consider the bigger picture: value. And that is why it makes more sense to measure the outcomes of your work. Are customers happy? Do they pay for what you create? Do they return? In this post, we shared some examples. Not just for pizzerias, but also for software development. Give it a try, and see what happens!