Engagement Series
User Engagement — First-Time User Experience
The hard truth about First-Time User Experience
User Engagement is a challenging task for almost all product teams. To keep users engaged the product team needs to complete 3 steps:
- figure out users' needs and preferences
- deliver the first portion of value to users
- continue delivering value to users
First-Time User Experience (FTUE) is about completing the first two steps.
Often product teams think they have plenty of time to do this. But the reality is very different. 👇
How many users return to the product after the 1st session?
Below is a chart showing user retention of the daily signup cohorts.
From the chart above we can learn the following hard truth:
- in the worst cohort, only ~25% of users returned to the 2nd session
- in the best cohort, only ~40% of users returned to the 2nd session
It means that more than half of acquired users didn’t get into the product and decided to leave forever.
The most obvious reason for these low figures is a combination of two factors:
- bad user targeting
- bad FTUE
What time does the product team have to help users to get into the product in the 1st session?
Let’s build a density diagram of the time spent by users with the product during the 1st session.
From the chart above we can learn that user drop-offs in the 1st session are multimodal:
- the 1st peak is in 0 seconds (technical issues?)
- the 2nd peak is around 10 seconds (targeting or onboarding issue?)
- the 3rd peak is around 110 seconds (onboarding or value issue?)
As we can see, the product team has very tough time constraints to help users get into the product in the 1st session.
Half of the acquired users spent less than 6 minutes on the product in the 1st session. Some of them will churn within first 10 seconds.
This means keeping the onboarding process simple and containing only steps that could improve the user experience within minutes.
Can hyper-short sessions be considered unsuccessful?
To answer this question let’s calculate IV / WOE for 1st session time bins. I am going to bin 1st session time by deciles (10% of data per bin).
The approach first calculates shares per bin for each case (returned/not), then calculates the Weight Of Evidence per bin, and finally calculates Information Value.
The first thing that I want to mention is that IV = 0.26. Regarding IV, this signal has medium predictive power and is good. From my experience, a lot of signals could have a lower predictive power.
The more interesting moment here is the relationship between 1st session time and whether the user returns to the 2nd session within 24 hours:
- if the 1st session length is up to 40 seconds then a user is more likely to return than not
- if the 1st session length is from 40 up to 1138 seconds then a user rather not return than return
- if the 1st session length is more than 1138 seconds (19 minutes) then a user is more likely to return than not
So as we can see the relationship between 1st user session time and the likelihood of returning to the 2nd session is not linear.
A short first session is not necessarily a bad one, and a long first session is not always a good one. Context is very important.
How long does it take for users to return for the 2nd session after signing up?
Let’s build a chart to show what cumulative number of users return to the product in X hours.
The figures are quite interesting:
- in 8 hours up to 48% of users who have 2nd session return to the product
- in 24 hours up to 73% of users who have 2nd session return to the product
- in 48 hours up to 84% of users who have 2nd session return to the product
If the user doesn’t return to the product in 24 hours we should start worrying about him and react to this.
Another important conclusion from the chart above is that some users will return to the product in some time:
- 16% of users who have 2nd session will return to the product in 48+ hours (2+ days)
- 4% of users who have 2nd session will return to the product in 168+ hours (7+ days)
If we continue to push one-session users to return to the product some of them could actually return.
Could we use the percentage of users who returned to the 2nd session within 24 hours as a proxy metric for the quality of user acquisition targeting?
As I have already shown in the previous post, if users don’t return to the product within 24 hours then only 4.8% of them will return to the product in the next 24 hours.
That’s why I believe this metric can be used as a proxy metric for the quality of user acquisition targeting.
Let’s dig deeper into the first milestone of FTUE — D1 retention.
