Analytics for Product Managers : What to measure, When and Why? Part 1
“There are two possible outcomes: if the result confirms the hypothesis, then you’ve made a measurement. If the result is contrary to the hypothesis, then you’ve made a discovery”
- Enrico Fermi, Nobel Prize winner in Physics
In the process of building successful products a product manager normally steers the product through multiple iterations. Very rarely one gets to see products which take off and find acceptance with consumers as soon as they are built. One of the important tasks during product iterations is the ability to measure the impact of the product on the customers. This should be done by measuring how the customers have been adopting the product, identifying the gaps in the product based on the data that is measured and trying to figure out the ‘why’ through user research qualitatively.
In new product development one might have to loop through that cycle multiple times before one can get a significant traction among the users.During the iterations the question that inevitably pops is whether one is measuring the customer data and metrics in the right way. With the deluge of information that is available, how does one know? What are the metrics that one should track to figure out whether one is in the right direction in meeting the business goals. Does it depend on the type of business or on the stage that the business is in and the business model? Compounding the problems is the fact that as a product manager you will have strong preconceptions about how the users think and this can steer your decision making in a wrong direction.
This is where a strong set of metrics and deep dive analytics come into picture. Measuring something forces you to be accountable. Metrics and KPIs are systematic guides that help you to address all your assumptions in the business model. They force you to keep track of the question you have about your product, while keeping your assumptions about the business model in your mind. They focus your attention on the critical issues and most efficiently employ your time and resources to remove the biggest risks. And they also provide a systematic way to respond to the real life data that users generate while using the product.
This post is an attempt to make sure that one can track the metrics and KPIs in a systematic manner while watching out for signposts and directions to change them depending on the outcomes.
Analytics in the Lean Start-up Framework
Before defining metrics, it is a good idea to understand what the main aim of analytics is. In order to do this it is recommended to understand analytics in the lean startup framework because Lean Startup is an optimal methodology designed to get companies to arrive at a product that truly serves the customers’ needs. This is done by iterating the product multiple times through the build-measure-learn cycle. The teams build a prototype of a new product, measure how it performs, and learn from the experiment. The goal is to go through this cycle quickly to maximize learning and reduce costs in a short amount of time. The end result, ideally, is a more effective and agile company.
Contrasted with “ build and they will come” waterfall methodology of building products, lean startup focuses on collecting continuous feedback (quantitative as well as qualitative) from the customers and include that feedback into every iteration so that the product can be steered towards a product-market fit much faster.
Eric Ries postulated the lean startup process when he combined customer development, agile software development methodology and lean manufacturing process into a framework for developing products quickly and optimally.
In every iteration of the build-measure-learn cycle, analytics will help you clearly see whether you are moving towards the goals you had set for yourself. Once you know what your business goals are, you’ll then need measurements to know if you’re making progress towards your goals.
The kind of metrics you measure depends on what stage the business is in and the type of business you are in.
Why can’t I just use financial metrics like most of the established companies?
There are multiple reasons why you can’t use established financial metrics to measure the progress of new products: For starters in the early stages of the product, your revenue and cash flow both might be zero. So all the metrics and ratios fail miserably in capturing the progress you are making.
Secondly, most traditional management tools are designed for execution and not for exploration, for optimizing a known problem and not for discovering a new problem to solve. Using the wrong set of metrics at this time will make you waste a huge amount of effort and lead you on the wrong path that might lead to failure.
Here and Here Nathan Furr@ Inc.com explains “You can’t run a discounted cash flow analysis (DCF) because you can’t predict the cash flows under uncertainty and calculating a return on investment is not just an exercise in fiction, it is likely going to damn your innovation to failure. Why? Because you are using metrics appropriate to measure a relatively certain business initiative to measure one full of uncertainty. As a result, you will set expectations for success that you can’t possibly meet and ultimately your investors and supporters will back out when you miss those predictions.”
Watch the video here which drives the same point home:
The primary aim of a new business is to find a scalable and repeatable business model. Steve Blank, in this article explains: “The second time you’ll need to know about Income Statements, Balance Sheets and Cash Flow Statements is after you’ve found your repeatable and profitable business model. You’ll then use these documents to run your business and monitor your company’s financial health as you execute your business model.”
Kosta Stavreas, explains in this article when is it appropriate to use your financial modelling techniques to measure and plan for progress. He says “Once a company has achieved product-market fit, is profitable and growing — a financial model will come into it’s own. By this stage, the company will have lots of market fact upon which to base some projections. They will have past trends in sales performance, an understanding of their customers needs, increasing volumes of attention and feedback in the market, and an innate understanding of what they must to do satisfy their customers and grow.
Arguably, this is the time for doing “detailed numbers”. The company has market facts which it can use to create sound forecasts (at least for 12 months) and can manage the company’s operations through a wide range of metrics. Growth can be paid for with the cash the company is generating and how well the company is executing can be measured.”
What are the characteristics of a good metric?
In his book, Lean Analytics, Ben Yoskovitz defines analytics as the measure of movement towards business goals. Once you know what your business goals are, you’ll then need measurements to know if you’re making progress towards your goals.
A good metric should be comparative. A 5% conversion rate is good to know. But it wont tell you much if it is not compared against anything. What was the conversion rate last week? Are your conversions increasing when compared to last week? Metrics which answer questions like those will be more useful than just absolute numbers.
It must also be understandable and It shouldn’t be complicated, everyone should understand what the metric measures and be able to focus their efforts towards measuring and improving that metric.
The metric should also be a ratio or a rate. Absolute numbers such as line number of hits, number of page views etc. are good to know but the percentage of monthly active users returning to the site — that is a great way to measure engagement.
Good metrics change how you behave. If there is a change in the metric and by looking at the change you are not sure what to do with that information, then it is a bad metric. If it doesn’t change how you behave, it’s a bad metric.
Types of Metrics
Actionable Metrics : Actionable metrics change the way you behave by helping you pick a course of action.
Eric Ries in this article explains: “The only metrics that entrepreneurs should invest energy in collecting are those that help them make decisions. Unfortunately, the majority of data available in off-the-shelf analytics packages are what I call Vanity Metrics. They might make you feel good, but they don’t offer clear guidance for what to do.”
This article at Kissmetrics explains why vanity metrics are all those data points that make us feel good if they go up but don’t help us make decisions.
Exploratory Vs Reporting Metrics:
Exploratory metrics are required to find unknown insights from the data while reporting metrics help you give visibility into day to day operations.
Here is an example of exploratory data analysis of Facebook page analytics.
Leading Vs Lagging Metrics:
Leading Metrics give you an understanding of what is going to happen next. Lagging Metrics explain what happened in the past. Leading metrics are better because you still have time to act on them.
Using Mobile gaming as an example, Matt Ho explains how a leading metric is an indicator of the potential of your business. It predicts the future.
Correlated Vs Causal Metrics:
If two metrics change together then they are correlated, but if one metric causes the other to change, then they are causal.
Correlation and causation are two of the most important concepts to understand if you want to create growth. Ben Yoskovitz, explains the difference between correlation and causation by stating “correlation helps you predict the future, because it gives you an indication of what’s going to happen. Causality lets you change the future.” Knowing the difference between the two goes a long way in ensuring that your business decisions are based on hard facts and measurable variables.
How do I know what stage my company/Product is and what metrics to apply at those stages?
There are four stages that a new product goes through in its lifecycle. These include the Discovery, Validation, Efficiency and Scale. Based on the stage the product/company is in you can measure the metrics accordingly:
Discovery & Validation: Irrespective of how confident you feel about a product idea, you should always invest time and effort to figure out whether the problem is worth solving. The best option for this is to speak to the users directly. The focus is on qualitative validation of your idea.
Jon Lay and Zsolt Kocsmarszky at the smashing magazine describe a systematic and 4 step process of going about it: 1) Validate the problem 2)Validate the market 3) validate the product 4)Validate the willingness to pay.
Andreas Klinger has an excellent presentation here on the metrics during this phase.
Apart from the qualitative data you have from the user interviews, the most important metrics you need to measure from your product (or MVP) during this phase are activation and retention metrics. Primarily, during this phase you are interested in knowing how the users are interacting with the product and how valuable do they find the product to be. These are based on the framework developed by Dave Mclure’s Pirate Metrics. .
This video here explains why during this phase Activation and Retention metrics are are the metrics that matter.
Kissmetrics in the blog about how to get new users to become paying customers defines activation as: The first point where you deliver the value that you promised.
Dan Wolchonok @ Hubspot describes what activation is. With great examples he describes why this is such an important and a very actionable metric.
In this article at Intercom, Clement Delangue, head of marketing at Mention says “As always, everything starts by picking the most relevant metrics to track. The activation rate of our key converting features was the metric that we chose to assess and improve on at mention.”
Drew Mckinney explains in this article :” Feature retention is usually measured through cohort analysis. At its most basic, cohort analysis is a behavioural analysis tool for observing how groups of people (“cohorts”) experience something, usually over time.
For the purposes of user retention, cohort analysis answers the question “do people come back to use this again and again?” That is, do people who viewed this feature last week continue to come back and use it today? Do they do use it every day? What percentage drop off over time? Retention tells you how valuable something is to your users.”
Jonathan Hsu, Partner @Social capital describes how to analyse engagement and retention for apps/products that intend to grow users and monetize them in the future
This article by amplitude describes 3 way to measure retention: N-day retention., unbounded retention and bracket retention based on the type of app.
This article by amplitude describes 5 important questions you need to ask of your data so you can evaluate your retention. This will give you all the information you need to start building a retention process and start improving your app for all your users.
PS: The remaining part of the article is rolled over to Part 2. Part 2 will include metrics for efficiency and scale stages of business and metrics based on the type of business model(ex: e-commerce, SaaS etc). Keep watching this space :)
Originally published at IndiPM.