Aug 4, 2016
How to measure product retention
If you have any experience working on a digital product, you’ve seen your fair share of metrics. Metrics give us a better understanding of who is using your product and how they’re doing it. Metrics also let your team make actionable choices based on this data.
The go-to metric for most teams is overall usage, which gives you a view of feature usage by the numbers over time. Even if you are not on the product team, you’ve probably seen usage metrics paraded around not long after a feature launch.
The risk with only looking basic usage metrics is that basic usage metrics don’t tell you how people are using your feature. You might see great results from a feature launch at first (“look how many people are using this!”) followed by a steady decline in use weeks later. What happened? Usage won’t tell you, but retention can give you deeper insight into how people are using your feature, and let us make choices about how we should proceed.
Retention tells you how valuable a feature is to your users
Feature retention is usually measured through cohort analysis. At its most basic, cohort analysis is a behavioral analysis tool for observing how groups of people (“cohorts”) experience something, usually over time.
For the purposes of user retention, cohort analysis answers the question “do people come back to use this again and again?” That is, do people who viewed this feature last week continue to come back and use it today? Do they do use it every day? What percentage drop off over time? Retention tells you how valuable something is to your users.
Cohort analysis tools are present in every modern metrics tool. Google Analytics, Mixpanel, and Kissmetrics all have cohort analysis (or “retention”) charts. But you rarely see these in presentations, and I suspect it’s because they are misunderstood.
How to read cohort analysis (retention) charts
Cohort analysis charts can look intimidating at first, but once you understand them, you know exactly what to look for.
Check out the example below. I’ve taken a row from the above retention chart produced from Mixpanel. What I can see is that from the cohort who started using the product on Jan 11, 40.35% returned the subsequent week to use the feature again. The week after that, 42.39% of the original Jan 11 cohort returned. By week 9, 25.35% of the original cohort was still around using the feature.
Use cohort analysis to define the value of a feature
The best measure of a feature’s success is found in it’s “long tail” retention. This tells you about the continued use of the feature over time. For practical purposes, this means looking several days, weeks, or whenever you see usage “evening out” around a specific percentage.
If you see a weak long tail (less than 20%), this is a sign that most users are not returning, and presumably not getting any value from the feature. For your team, this could mean that the feature might not be worth pursuing any further, or that major adjustments need to be made to improve it.
Anything between 20% and 30% should be reassuring that the feature has value to some of your users, but perhaps a sign that it’s a niche feature, or not valuable enough for most to return. Anything above a 50% retention can be considered a success.
If you have them available to you, start looking at your retention metrics. They give you a lot more depth into feature’s value vs. simple usage metrics, and can be indicators of what you should do next with your product.
