Engagement Series

User Engagement — key events

To get users engaged you need to activate them first. Key events are the shortest path to achieve this.

In one of my posts on LinkedIn, I stated that products often lose most of their new users in the 1st session.

It’s easy to prove this fact by building a chart like this:

User cohorts retention by session number.

From the chart above, we can learn that only 25% to 40% of users return for the 2nd session. It means that 60% to 75% of users just churned during the 1st session.

But the 1st session is not the only issue. In another post, I proposed to decompose the metric [% users who returned on D1] into the following two metrics:

• [% users who started 2nd session on D0] and
• [% users who returned on D1 after 2nd session]

The latter metric as a rule has a higher percentage than the former and that’s why it is mathematically more important.

So, while the 1st/2nd/3rd sessions are important, what users did in the product at that time is even more important and determines whether users return the next day.

Key Events Definition

Every product consists of many product features. Some product features are exposed to users only once — as a rule, it’s onboarding flow. Other product features are available to users permanently.

Depending on product feature complexity product feature can fire one or several events while users use it. To simplify our analysis let’s assume that each product feature fires just 1 event per one usage.

Importance

To determine which events are key, we must define how to measure their importance. A product feature is important if a user returns in the next period after using it.

There are two approaches to calculating retention:

• based on users returned to the product
• based on users returned to specific product feature

I recommend using the approach “based on users returned to the product”.

Survival bias

The next important moment is to understand the following:

• some users use a specific product feature and return (that’s how the retention rate is calculated, but there are a few more cases 👇)
• some users use a specific product feature and don’t return
• some users don’t use a specific product feature and still return
• some users don’t use a specific product feature and don’t return

We can easily overestimate the importance of a particular product feature if we only consider its user retention. To do a more realistic assessment we need to take into account all 4 cases mentioned above.

I tested several approaches and concluded that the MCC coefficient is the most appropriate: it’s easily calculated and works well with imbalanced data which often happens with product usage data.

Key Events Identification

In practice, it’s really difficult to determine which session sequence number impacts user usage the most.

Some users are quite fast and start fully using the product in the 1st session. Other users are more concerned about whether the product fits their needs. They do a few tries before making the final decision.

That’s why I recommend taking a step back and using the whole D0 as train data and retention on D1 as a target.

Let’s take users who used the product on D0, get their usage and D1 retention stats, calculate the MCC (prd) coefficient, and aggregate these data on the product feature level.

The resulting table could look like this:

The MCC coefficients for product features.

As we can see different product features have different popularity, D1 retention rate, and magnitude of the MCC coefficient.

I created the following chart to represent product features like a funnel:

Funnel and MCC coefficient

There are a few interesting insights here:

• Some product features (mostly from onboarding flow, at the top of the funnel) have a negative impact on user retention
• Product features that impact retention the most are in the middle of the funnel

I like that the MCC coefficient distinguishes between positive and negative impacts on user retention. Also, it’s great that the MCC coefficient considers sample size.

Our data suggested that two product features that impact the most user retention on D1 are:

• feature32
• feature28

Both features are related to push notifications. Their impact on user retention makes a lot of sense to me.

If users have enabled push notifications and interacted with them during their first few sessions, then:

1. users likely have already received some value from the product
2. users have already set up the product to get additional value from it in the future

Key Events vs Core Actions

Some of you would probably want to ask me: are these push notification events the most important thing that the product manager should focus on D0?

To answer this important question we need to clearly distinguish key events and core actions:

• core actions are a few global actions that the product was designed for. For example, Uber’s core action could be a request for a ride. It’s the action that delivers the main product value to users.
• key events are local actions that can help users move forward and get more value from the product. In the case of Uber, allowing access to user locations and push notifications could be that kind of product feature. They are prerequisites to the main value.

So, key events don’t substitute core actions. They are micro-steps that maximize the success of getting value from the product and actually, after a few periods of analysis, key events that you identify will be nothing else as core actions.

Key Events Verification

It’s always a good idea to test your insights against your data.

The simplest approach is to test the relationship between users exposed to these key events and their retention on D1.

% users (with ftr 32|28) vs % returned users prd (p1).

From the chart above we learn that there is a linear relationship:

• the more users in the cohort were exposed to key events on D0, the higher D1 retention for that cohort.
• R2 (linear regression fit measure) = 0.5 and it is quite high for such type of analysis. If you read my previous post about different signals on D0 you can figure out that this signal is one of the strongest ones.

Key Events Stability

The most concerning point is that the engagement with key events in user cohorts varies greatly (from 14% up to 26%).

Two fundamental factors that influence product usage constantly are:

• experimentation with User Acquisition
• experimentation with Product

We can’t freeze User Acquisition or Product development to have a more stable environment and as a result model with a better fit.

However, we can learn from data how often our model fits data well.

Product feature MCC (prd) coefficients by date.

From the chart above we can learn that the MCC (prd) coefficient for product feature32 (our most impactful feature) varies a lot:

• 0.1068 on 05/10 (minimal value)
• 0.3367 on 05/19 (maximum value)
• 0.2132 — weighted average value for a whole period

For sure it’s always better to have the MCC coefficient as high as possible. But as you can see from the data to get high coefficients at the beginning of product usage is unlikely.

Nevertheless, we can rank product features by MCC coefficient for each day. This could help us to more easily compare the importance of key events on different days.

Product features ranked by MCC (prd) coefficient by date.

From the chart above, we can learn that:

• product feature32 had ranked first in 9 out of 18 times. Also, this product feature was in the top 2 in 13 out of 19 times.
• product feature28 had ranked second in 5 out of 18 times. Also, this product feature was in the top 2 in 13 out of 19 times.

So we can be quite confident that key events behavior is repeatable.

Product Overview

It’s always useful to build a scatter chart and learn from all product feature positions.

Product features map.

There is a very limited number of product features that could heavily impact user retention on D1 and probably this is the main result of this research.

From the chart above we can learn:

• 3 product features have the highest impact on D1 user retention (dark blue color dots)
• 10 product features have a moderate impact on D1 user retention (light blue color dots)
• 32 product features have a rather low impact on D1 user retention (light/dark red color dots)

That’s why to design successful Activation you should focus your users just on a few well-chosen product features at every stage.

Similar to dating: you don’t need to demonstrate all your qualities to impress your counterpart. Just be positive and calm and make sure you have a chance to go on a second date.

What’s next?

At different stages, different key events will determine whether users return to the product. It means that you need to create a series of such analyses and build a value chain map for your users:

• key events for D0 users that impact D1 retention
• key events for D1 users that impact D2 retention
• key events for D2 users that impact D3 retention
• key events for W0 users that impact W1 retention
• key events for W1 users that impact W2 retention

By doing this you will have information about actual users' behavior and signals that correlate the most with user retention in the next stages.

This information will serve as a reference for you in answering questions such as:

• which events create value and when
• what time does it take to activate users
• whether there are some shifts in core actions over time