How to ensure you are tracking the right product metrics?

the truth behind product analytics

As a product manager our goal is to deliver the most value to our customers, and we do so by looking at metrics — pieces of quantifiable data that illustrate the changes in revenue and customer behavior. The real power lies in how we learn from them, how we interpret, hypothesize, and ignite change.

Photo by : mohamed_hassan(pixabay.com)

In order to unleash the power of metrics, they have to be defined , designed, and visualized to detect problems, set new goals , spark questions, and take informed decisions. The right metrics can be built by ensuring a few checks and balances that can be introduced in the process of defining them. Let’s dive into some of the guiding principles to avoid the common blunders that come with defining metrics.

1. Actionable metrics with context (Understanding the “WHY”) — Metrics without strategy and context are just numbers. A metric reflects strategy and helps answers if they are working or not. The best way to avoid this blunder is by asking questions like — What are the key hypotheses of the strategy? What metrics would indicate those hypotheses are true? Sometimes teams focus on building dashboards because data is available instead of understanding why the specific data is relevant and how it can be actioned. For example, just defining retention rate is meaningless but retention rate over a particular time frame, user group is actionable. The best way to avoid this blunder is by asking questions like — Does the feature retain these people as its users? What is the right retention event (visiting/transaction) ? What is the time frame of measurement? Which event leads to churn? What is a good or bad retention rate? How does retention rate change over different customer segments?
2. User journey based events and metrics — It is important to understand the customer journey , define events of Intent → Success → Failure to understand the nature of user behavior. For example, for a payment app these events could be, Intent: Add New Payment Method Selected & Add New Payment Details Submitted . Success: Add New Payment Method Successful, Failure: Add New Payment Method Failed. Furthermore, understanding the difference between what an implicit and explicit success/failure event is is necessary. For example — a user drop off is an implicit failure while an order cancellation by a user is an explicit failure. Identifying and measuring these are necessary to answer and action different areas of user journey.
3. Input over output metrics — Most north star metrics and strategy are built using output metrics like a retention metric or revenue metric . However, to deliver these output metrics, the teams goal has to break them into input metrics which can be controlled and actioned. Setting input goals helps you understand the levers that drive our output. For example — Focusing on improving usage which leads to revenue .The best way to focus on inputs for the team is by using the OKR framework for each team which provides a simple way to create alignment and engagement around measurable goals which focusses on input metrics while delivering the inevitable output metrics.
4. Understanding depth of Engagement — Understanding and segmenting for engagement is important. While retention is binary , Engagement is depth. It answers the question, how active were they within the defined timed period? 0→N. Engagement is one of the major inputs into driving retention. The best way to understand this is by asking questions like — What are the events which define engagement for the product? What is the engagement rate for each event (impressions, views, shares, comments)? What is the time frame for measurement? Segmenting user groups/ creating personas based on depth of engagement (i.e. shares>comments>likes)?
5. Auditing and shifting based on user maturity — Over time, the user maturity shifts and so should the metrics. For a food delivery app like Gojek , the users original value prop was fulfillment. The user would know what to order and then the app was used to fulfill the order delivery. However, with time this has matured, the most prevalent user intent journeys shifted and started using Gojek as a means of new restaurant discovery rather than fulfillment. From knowing what to eat and using the system to deliver exactly what they want. Now, intent events would be things like scrolling through their friends’ food feeds, browsing discounted deals, or using the Nearby feature. Hence, it is important for users to track intent and change in intent to ensure the right product value is delivered. Some of the ways to do it is by asking questions like — How have the habits of my customer base changed? How might they change going forward? How has usage of my customer base changed or how is it changing ? Which input (Activation, Engagement, Resurrection) should I invest resources to improve?
6. Power of Adjacent user group — Once we have hypotheses for successful user segments and why they are successful, we can hypothesize possible adjacent users’ segments by changing one or more of the vectors. For long term success , it important to repeatedly find the next loop/adjacent user group. The data will help identify places in the product that people are dropping off which are ways to differentiate adjacent user groups. This is the starting point to help develop hypotheses about why different segments of users are dropping off. Constant research and hypothesis/experiments with small features will help identify new user groups, behavior and how to push users from one adjacent group to the other . When this is followed with conviction , making a value prop for a segment that is different on multiple attributes becomes a question of when and not if.

In this article, I have tried to address some of the big drivers which are key to define metrics for delivering product success. However, there are a few implicit practices for metrics like ensuring a shared terminology across teams, shared source of truth , trust in the way data is calculated and ability to act quickly on data which can save a lot of churn amongst teams & stakeholders.

Additionally, one of my favorite best practice which has worked over the years is :periodic evaluation of the fundamentals Of Registration, Activation, Engagement and Monetization and thus helps in refreshing the data elements and ensuring the alignment with the overall product goals. What are some of your typical ways to define product metrics ?