Notes from Inc42Plus Maker’s Summit 2021–Fix The Leaky Bucket: Taming The Retention Beast

A masterclass by Brijesh Bharadwaj

You can find the original video on Inc42’s website.

Outline

• How to focus on and identify right metrics when working on user retention
• How to run quick a/b tests on multiple user groups to drive user retention
• Importance of UX in User retention

Why should I care about retention?

• Affects everything growth related

A Higher retention means:

• Faster user acquisition: More new users retained, faster.
• More monetization: The ability to spend more on paid acquisition.
• Faster recovery of marketing spend.

A higher retention results in faster user acquistion, more monetization and a faster recovery of marketing spending.

Defining your retention metric

• What is the frequency of the problem/need your service solves? = Problem frequency
• What is the event in your service that indicates that your service solved that problem? = Value event

Retention metric = Problem Frequency + Value Event

Problem frequency and value event

• Ask your users
• Look at your data
• Use your intuition/judgment
• Each use case has a different value event and problem frequency

E.g. 1:

• Service: Dunzo
• Problem: Ran out of vegetables and fruits.
• Problem Frequency: Once/twice a week
• Value event: Completed a vegetables and fruits order (At what point in the service do you feel like we’ve delivered the value to you?)

E.g. 2:

• Service: Instagram
• Problem: I want to know what my friends are up to
• Problem Frequency: Multiple times a day
• Value event: Viewed a story/post

Validating value event

• If you are confused about what is your value event, you can check data to see the cohort of new users who did that event and see which cohort has a higher retention
• This will be a balancing act of ensuring a large enough cohort of new users do this and the step jump in retention you get as you go deeper in the funnel.
• Kindle App: Is the value a user buying a book, or is it a user completed a book or user completing a book and rating it 4+?
• If the delta is large, pick the event that has a higher retention.
• If the delta is small, pick the event that is done by more users.

Validating Problem Frequency

E.g.

Fig 1: Retention of users on Dunzo

• The frequency with which Dunzo users are coming back to the platform
• Users seem to be coming back after a week — this seems to be the problem frequency as according to Fig 1.

Defining your retention metric

• Guided by your problem frequency and value event
• Daily (problem frequency guided) active (users who perform the value event) users

Analyzing Retention

What is good retention?

• Some portions of the users are staying on the app “forever”
• Too many users shouldn’t be dropping off
• The number depends on the business model
• What is the retention I need to be able to make my business work?
• E.g. Adding 1000 users per month with 45% M12 retention and 8% M24 retention
• Check for the deviation from the average retention — can give you outliers
• Very high positive deviations are good, very high negative deviations are poor = analyze and understand why either of these happened
• Is a full refund a sustainable user retention strategy? → Depends on the userbase. Can A/B test this. Need to test the impact on retention and whether it’s financially feasible over the long run.

Improving retention

• Use user attributes to identify where the skew in retention is happening from
• User demographics like age, gender, have kids/no kids, have pets/no pets, etc.
• Look at the existing competition and look at unsolved problems — ask yourself: why aren’t people solving these problems? Can I solve these differently? (Require a differentiation factor)

Ask yourself, does this meaningfully change the way the user uses my product?

Some questions to ask while considering user attributes:

• All things equal, can a user’s income affect the way someone uses a delivery app?
• Does this person value convenience or money more?
• Does the person’s attributes impact the frequency they order?
• Device (iOS/Android) — a proxy for income or a specific device version can be performing poorly
• Geography — a proxy for income/service quality/user attributes
• Product category — healthy food orders vs pizza food orders vs Chinese food orders
• Product feature usage — first order through search vs through categories; first order placed via COD vs UPI
• Once you have an understanding of what is causing better retention, you can run experiments to skew towards attributes that are known to give you more retention
• E.g. if you have a better % of M12/signups with users who are living in apartments with rent > 25k/month. Then running branding activities in similar apartments is a valid experiment.
• E.g. if say having a pay later option results in a higher % of M12/Activations then increasing the number of users who have a pay later option is a valid experiment.

Difficulties in measuring retention

• Running a long-term experiment on retention is tough — have to wait for a long time and jumps in retention are very small.
• Require a proxy for retention — if this happens, then retention is also going to happen.
• E.g. of a retention proxy: Users have a higher likelihood of retaining when they use the product a certain number of times (X) within a certain period (Y) of signing up.
• Can discover X and Y using a correlation analysis + can take an accurate guess based on a problem frequency
• Fig 2 shows hypotheses with the correlation. Can also use other metrics such as Positive Predictive Value (PPV) and Negative Predictive Value (NPV).
• PPV = (This event happened AND user retained)/(This event happened)
• NPV = (This event did not happen AND user did not retain)/(This event did not happen)

Fig 2: Calculating the correlation of hypotheses and M12 retention