How to measure the success of a Product Feature?

by Arpit Agarwal — Vice President (Analytics & Data Science) at Khatabook

A successful product is simply an offering that solves some problem for customers and make their life easier. Isn’t it?

Look around you, you are surrounded by products! Right from the paint on your wall, to the ceiling fan, the clothes you wear and the mobile/ laptop you are using to read this article.

Making a successful product is a long-drawn journey. Even if you hit your luck to be successful the first time itself, it would be copied faster than you can imagine. So, our battle is not against others. Our battle is against ourselves to keep improving, outpace others and be the best version of ourselves.

“The only reason why people will choose your product over others is because you are solving the problem in a better way at a better cost!”

One of the common mistake product managers make is “Getting emotionally attached to the features they developed”. Due to this, the focus shifts from solving the problem to saving a suboptimal product feature from failing.

Hence, Step 0 is “Learn to let go” and “It’s fine to fail”

Now, let us delve deeper to define the success of a product feature.

Step 1: Define the problem you want to solve and have a clear objective

Let’s take Facebook as an example:

Problem Statement: To connect people with each other seamlessly!

Most of the product features can be classified to have 4 larger objectives.

1. Acquisition — When we want to increase product installs

(e.g. Send Invites to contact list to install the app)

2. Retention/ Increase Usage — When want customers to stick to the app/ features

(e.g. Facebook Core features like ‘Photos’, ‘add a friend’, ‘messages’ etc)

3. Engagement — To increase the top of the funnel from existing / dormant users

(e.g Notifications on FB whenever someone posts on your wall or mentions you, ‘News feed’ etc.)

4. Monetisation- Increase conversion to subscription/ orders etc

Sometimes, we make a mistake to attribute more than one objective to a feature. It is best to have only one of the above as primary objectives. If the feature does not live upto expectations of primary objective, it is considered as Failed!

Step2: Evaluate your features through the 8-point Success framework

1. Adoption: Is the feature being used as much as we’d like it to be used? Who are more likely to adopt this feature?

KPI’s: % users using feature, time to adoption, organic vs assisted adoption etc

2. Cost: How much are we spending to adopt this feature? Are the cost justifying the adoption?

KPI’s: Cost per adopted user, Cost per retained user

3. Discoverability — How is the feature discovered and what is the customer effort associated with it?

KPIs: No of clicks to use a feature, Time spent prior using feature etc

4. Usage — How frequently are users using the feature?

KPIs: No. of times used per session/ day/ time fo the day/ month etc

5. App Impact — What is the impact on app engagement, and other key app features?

KPIs: App engagement in Control vs Test set, Cross feature usage in Control vs test etc

6. Retention — How many customers are repeat users of this feature and does feature also impact overall product retention?

KPIs: Retention of feature, Retention of product

7. Customer Satisfaction — What is our customers’ overall sentiment towards this feature?

KPIs: NPS score, CSAT score etc

8. Health Metrics — Is the product available & performing in the manner that users would reasonably expect?

KPIs: Latency, drop rates, time to load, errors etc

This framework will clearly tell you in what aspects your feature is performing well and where it needs improvement.

Step 3: Define Success!!

The success measurement depends on the problem you were solving.

a. Say you wanted to increase your Engagement (say DAU) by 10% then the engagement feature should help bridge that gap significantly

b. If the Objective was Acquisition then it should be judged basis no. of acquisitions and their quality. Also, the cost of acquisition should be lower than your other performance/ brand marketing alternatives

c. If the Objective was to increase Product usage/ Retention then product retention should increase significantly

d. The same goes for Monetisation…..

If your product feature goals are met with no negative impact on user experience, the product feature can be classified as successful.

Lastly, remember the mother of all success mantras…

“Don’t find customers for your product, find product for your customers”