Differentiating Metrics: Product vs Feature
In 2019, I helped organize the Product Management Festival conference in Zurich, Switzerland. One of the highlights for me wasn’t the conference itself, but the chance to speak with Netflix’s ex-VP of Product Gibson Biddle, who was one of the keynote speakers. I asked him what metrics he paid the most attention to as VP of Product at Netflix. He explained that the CEO focused on growth while the CFO concentrated on the bottom line, and so as the VP of Product, he balanced them out by focusing on retention. Since then, I’ve been focusing on retention metrics for products.
As a product manager, how do we evaluate the performance of a product vs. just a feature? Each product has its own features. If a product is considered good, does that mean all its features are also good? We know the answer is NO.
Let’s say we have a performing product with ten features. Eight of them are good, but two aren’t. If we only see the performance of the product as a whole, not the performance of each individual feature, the two bad features would be hidden behind the good ones and end up as deadweights. We have to look into these underperforming features to fix or sunset them, and into the good features to be maintained or even doubled down.
So how do we know which features are good?
The often-used AARRR framework is better suited to measure a product’s overall performance, not its features’. AARRR is short for Acquisition, Activation, Retention, Referral, and Revenue. The Acquisition, Retention, and Revenue metrics are most frequently used to analyze a product. However, It is not always possible to measure the performance of an individual feature in terms of product-level metrics.
- Acquisition: This metric is useful for a product as a whole. It’s not easy to measure how many users of a product are acquired by a particular feature. For example, one of the data most often analyzed in the acquisition phase is the Customer Acquisition Cost (CAC), this is very complicated to measure for each feature.
- Retention: The overall product retention cannot be translated directly to its features’ retention because each feature in a product has differing usage frequency. For example, in an e-commerce app, mobile phone bills are charged monthly, while property taxes are billed yearly.
- Revenue: Most of the time, we would want to know how each feature contributes to revenue so that we can sort our priorities easily, clearly, and objectively. It’s a good intention, but in reality not all features contribute directly to revenue. If contribution to revenue becomes the only consideration in analyzing a feature’s success, then platform-related features would always be seen as lacking. Likewise, smaller features would also lose priority.
Evaluating features is challenging because it’s not always easy to directly attribute product outcomes like success, acquiring, retaining, or monetizing users to individual features. To effectively evaluate features, we must consider other metrics:
- Adoption: The number of users who try the feature for the first time. The total of newly acquired users of a product may not engage with all of its features, so it’s important to see the number of new users per feature. But this alone is not enough, because sometimes poor features could have a lot of users due to a high level of exposure and promotion. Similarly, the lack of exposure and promotion could make good features become less popular.
- Retention: The amount of users who repeatedly use the feature based on the natural usage frequency for the feature. This metric exists to balance the overexposure bias stated on the adoption metric. Inadequate, overpromoted features would net a lot of first-time users, but this number won’t last long as users would eventually churn and stop using the feature.
- Active Users: When the above metrics show good results, then the total active users of a feature would gradually increase over time.
After thoroughly analyzing the features in our products, we can come up with a stronger strategy based on actual data. If not, then our strategy would only be based on gut feeling. Good luck!