Measure what matters: how to align your analytics with your product vision

During World War II, researchers from the Center for Naval Analyses had come up with a data-driven approach to optimize where armor should be added on airplanes. They studied the distribution of bullet holes and damage on the aircrafts that had returned from missions and concluded that wherever planes were sustaining the most damage was where armor was most needed. Sounds pretty thorough, right? But Abraham Wald, a mathematician, shot down this conclusion right away. He pointed out that the study only looked at the planes that returned from their missions and it ignored the ones that never made it back as a result of the damage they sustained. So, in fact, the “missing bullet holes”, i.e. the areas on the plane where there were no bullet holes, were the best indicators of where armor was needed.

A data-driven approach to building your product is great — but only if you’re measuring the right things. “Data-driven” is often taken to mean that the business and product are driven by the metrics. Unfortunately, too often the metrics used are simply those that are easily measurable (e.g. registered users as a proxy for usage) or are popular to measure (e.g. Revenue instead of NPS when you haven’t yet proven product-market fit for your core feature set). Pirate metrics might be exactly the right kind of metrics for you to track, but they come packaged with certain assumptions about your business which may or may not be accurate.

What you choose to measure is an important strategic decision. Your metrics should tell you if you’re on track to achieve your vision, and how well your strategy is working. This decision is also strategic because measurement costs time and resources — both to build in the instrumentation into the product and then again when you have to analyze the data when it becomes available.

To measure the right things, you should align your measurement approach to your overall Product Strategy. How do you do that? Read on.

Measurement Strategy Canvas: how to decide what to measure

After you have created your Radical Product Strategy, have it next to you as you work out your Measurement Strategy.

There are 3 steps to filling out the Measurement strategy:

Step 1: Prioritize the elements of Product Strategy to be measured

Depending on the stage of the startup and your funding level, you may not be able to prove out all of these elements, so you’ll have to prioritize and pick wisely. In early-stage companies, the top priority in measurement is usually Real Pain Points — it’s pointless to prove out Design, Capability, and Logistics before you prove out the Real Pain Points, since all of the other elements serve to solve those pain points.

Assign priorities 2, 3 and 4 to the other elements, Design, Capability, and Logistics. For example, in a startup with a high technology risk, you would prioritize Capability over Design.

Step 2: Decide the goals of each product strategy element that you want to prove out

Let’s start with Real Pain Points. What questions would you want answered to validate the customer personas and their pain points you identified in your Product Strategy? Next, for each prioritized elements 2, 3, 4 above, look at your filled out Radical Product Strategy and identify the aspects that carry the most risk and would be the most beneficial to prove out. For example, for Capabilities, you might list the aspects of your new technology that are the riskiest, and without which your solution would fall short. Every time you eliminate risk, you know you’re closer to achieving your vision.

Step 3: Describe how you would measure each of the goals listed

What would you measure to prove each element in the previous step? What metrics would prove that the strategy is working and whether you’re getting closer to achieving your vision?

Tip: The simplest measurement strategy for the Real Pain Points could involve creating a solution that’s manual (e.g. using services or simple tools like spreadsheets that may be unsustainable long-term) just to see if the pain point exists and if you understand it clearly.

Make sure you’ve filled out the Measurement Strategy in detail for the Now column of the Product Strategy framework. You can get away with less detail for the Next and Later phases of your Product Strategy, since those may change. However, having at least an initial sense of what you’ll be measuring in the future will give you the ability to plan that into your product early and track progress over time.

Tip: Educate your team and investors. Don’t fall prey to setting your measurement strategy based on how investors or popular metrics might define “traction”. Think of measurement as burning down each of the risks in your business so you can prove to yourself that there is a market and that your approach at addressing this market is working. Educate your team and investors on why the metrics you selected are the best indicators of traction for your company.

This strategic approach to measuring data is also contrary to the philosophy of “we measure everything”. A tech company was recording terabytes of video every day with the goal of measuring everything. But since storage wasn’t infinite, they discarded videos every week. So when it was pointed out that a useful metric to look at a video time-series of a component over many weeks, it turned out that data wasn’t available. So, especially in storage-hungry measurement scenarios, unless you know what exactly you want to measure, you may not actually have access to it when you need it.

Even if your data doesn’t take up terabytes every day and you’re indeed able to measure everything, you still have to select the metrics you’ll work on improving. Trying to improve all metrics, whether or not they materially impact business goals, leads to a product disease we call “hypermetricemia”. The biggest symptom? Making many small changes to the product and iterating to optimize metrics, but not necessarily getting any closer to your vision.

If you enjoyed this article, please share to help others find it!

Product is a way of thinking. Radical Product is a movement of leaders creating vision-driven change — you can download the free Radical Product Toolkit, a step-by-step guide that makes it easy and practical to apply product thinking.