How regular are users using this feature per week?

Today I’m going to continue unpacking ideas about how to measure a feature’s success. And this time we will talk about such a thing as regularity.

If we want to succeed in product development we should focus our efforts on making a habit. And the critical measure of habit is repeatability. There could be two types of repeatability:

• N days repeatability (weekly habit pattern): how many days a user uses your product feature in a week?
• M times repeatability (daily intensity pattern): how many times a user uses your product feature in a day?

In this post, we will talk about the weekly habit pattern and the next one will be about the daily intensity pattern.

The first thing that I would like to emphasize is that before you start making any conclusion, please, make sure that your product feature is just working. If it’s not the case and users can't use it then the interpretation of the usage patterns could be misleading.

Provided that your product feature works well we can usually assume the more user uses your product feature — the more value user gets from it. That is to say, the more user uses a product feature — the better.

Before we start, let me remind you of our case from the previous post: we decided to improve chat visibility, so we changed the app home screen layout and did the chat button more visible.

As result, the metric [# users] started increasing from week to week.

chat feature: [# users] growth

It looks like the popularity of the chat feature is growing. Now we want to assess not only the popularity of the chat feature but also calculate its repeatability.

Let’s start by calculating the metric [# days per user].

To do this we need to:

• bin (group) our data by weeks
• within each week, for each user, calculate the number of unique days when a product feature was used by that user and average this figure among all users

The result of our calculation could look like this:

chat feature: [# users] vs [# days per user]

As we can see on the chart above the metric [# days per user] is declining.

2 questions popped up in our minds here:

• are these figures low?
• is it OK to have a downtrend?

To answer these questions we need to think for a moment.

The definition of our new metric: after calculating the real number of days’ usage per all users we average them. But by doing this we hide a pattern.

That’s why it’s much better to calculate the distribution of users who used a product feature for N days instead of compressing usage patterns to one number.

Let’s calculate such a distribution.

n_days_per_week distribution (absolute figures)

Can we make some meaningful conclusion now? Not yet.

We are scaling our product and that’s why [# users] is growing from week to week: 694–858–1089–1366. That’s why to assess the quality of such scaling it’s better to convert our figures to percentages.

Let’s switch from absolute figures to relative ones.

n_days_per_week distribution (relative figures)

Now we see the overall picture much better:

1. the figures of [# days per week] are close to 1 because on average 92% of users used the chat product feature ONLY for 1 day per week.
2. the downtrend appears because of % the 1-day users increases from week to week.

We managed to see more and explain current figures but whether we can trust these figures and conclusions?

Let’s continue thinking…

Users who started using our chat in a specific week should have signed up before the week starts ( [signup_dt] < [event_dt] ) to have a chance to use the product feature for up to 7 days a week. In other words, we should not count for users who signed up in the middle of a week.

Let’s filter out users who signed up in the middle of a week.

n_days_per_week distribution with dynamic filtering (absolute figures)

As soon we applied the filter for each week we get much fewer users who have enough time to use a product feature for up to 7 days a week.

Again let’s switch from absolute figures to relative ones.

n_days_per_week distribution with dynamic filtering (relative figures)

The situation and possible conclusions have changed significantly:

1. On average less than 80% of users used the chat product feature ONLY for 1 day per week. Before filtering it was 92%!
2. We see some decrease in [% users] started on the 3rd day of chat feature usage (figures are small so it’s better to continue tracking them).

Now our scaling does not look so terrible. Moreover, we have learned the real weekly habit pattern and when we need to start supporting users to continue using our chat feature.

SUMMARY:

1. Feature repeatability is very different from popularity.
2. Averages hide true feature usage, so look at the days' distribution.
3. Always filter users who joined the product in the middle of a week.

Next time we will try to answer the following question — how many times do users use your product feature in a day?