If You Can’t Measure It… Then All Is NOT Lost

Metrics, metrics, metrics! (Photo by Jinho Jung, CC 2.0 license)

There’s a well-known saying about metrics — “If you can’t measure it, you can’t manage it.” Let’s drill down.

There are two fundamental problems with this saying:

1. No one knows who said it. It’s sometimes attributed to Peter Drucker, sometimes to Tom Peters, sometimes to W. Edwards Deming. However, none of them said it. And what they did say typically contradicts the saying above.
2. It’s typically used to mean “if you can’t measure it, it’s not happening.” With the implication that a business process that doesn’t have a metric around is out of control and your workers are cheating you (great attitude!). This is usually not true.

As a product manager this concerns me, since metrics around product management are notoriously hard (i.e., impossible) to find and especially problematic, but everyone keeps trying to find them.

What makes these metrics hard (or impossible) to find? There’s a quote often attributed to Albert Einstein, but actually by William Bruce Cameron, that covers this:

“Not everything that can be counted counts, and not everything that counts can be counted”

And W. Edwards Deming said, in reference to the most important long-term issues for a company:

“The most important figures that one needs for management are unknown or unknowable, but successful management must nevertheless take account of them.”

These two very smart guys are saying there is important stuff going on in your company for which there are no metrics.

Nonetheless, people keep trying to find metrics for these important topics — they are important after all. And this is where problems arise. What makes a metric problematic? To explore this, let’s get to what Peter Drucker did say:

“What’s measured improves”

If you start to measure something, it doesn’t matter what, human nature will work to improve the measurement over time. The problem is that “improve” is a tricky concept for a lot of measurements. It might mean “make the measurement bigger over time,” (or “smaller”, or “more consistent” or any number of other potential improvements). But it’s much more difficult to associate a desired business outcome with those “improvements.”

Take product management as an example.

• I could measure the number of requirements that are written by the product management team. Because I’m measuring it, the product managers are going to start “improving” that metric. Which means… what? Write more requirements? Does that achieve a business goal, to have more requirements? What if I already have too many? Oh, I should improve this by reducing the number of requirements written? I am not going to achieve any business value by tracking this metric.
• Well then, let’s measure “the number of requirements are that implemented in the product.” Oops, I think I see another problem — the way to improve this measurement is to put a lot more requirements in the product — that means they will have to be smaller (less business value intrinsically), and the measurement does nothing to ensure that they are the most important ones for the customer.
• What about the number of revisions each requirement takes? Wouldn’t that be a good metric? If you care about the quality of requirements, then sure. If you care about the quality of the product, then you need to think again. Steve Jobs said:

“When you start looking at a problem and it seems really simple, you don’t really understand the complexity of the problem. Then you get into the problem, and you see that it’s really complicated, and you come up with all these convoluted solutions. That’s sort of the middle, and that’s where most people stop… But the really great person will keep on going and find the key, the underlying principle of the problem — and come up with an elegant, really beautiful solution that works.” (From Inside Steve’s Brain by Leander Kahney)

It might take three times through a requirement, or five times, or 10 times, to get a feature defined to the point where it accelerates sales and customer success. Do you want to incentivize your team to discover that value in the feature, or just to get features written that might not be as good as they could be?

In product management as well in many other important business processes:

If you can’t measure it, you’d better apply human insight to understand if it’s going well.

Not as pithy a quote, but much more accurate.

I write about product management at http://pmhardcore.com and podcast with Rob McGrorty at http://alltheresponsibility.com.