How popular is this feature among Weekly/Daily Active Users
Today I’m going to start unpacking ideas about how to measure a feature's success. And today we will talk about such a thing as popularity.
Let’s start with a tracking period. The two most widely used tracking periods are:
- month — is an excellent way to track overall progress and share it with the team.
- day — is a way to see what’s going on as soon as possible.
Nevertheless, for a lot of startups, the most optimal tracking period is not for a month or a day. It’s for a week.
On the one hand, in a week you could collect enough stats to make some reliable conclusions. In this post, I am going to use daily stats mostly for educational purposes.
On the other hand, you can adjust your stats to week seasonality which usually happens in a lot of products. In some screenshots below you can see regular spikes. It’s weekly seasonality in action.
Let’s assume you are developing some mobile app and inside the app, there is a chat.
You launched the chat feature some time ago and recently you decided to improve chat visibility as you believe that the chat is a great way to increase user engagement within the app. That’s why you changed the app home screen layout and improved the chat button visibility there.
Let’s start by checking [# events] — the number of events.
If we want to track user activities, then we need to trigger events whenever a user does some meaningful action (use a feature) in the product.
So tracking the number of events [# events] is the simplest way to assess a feature usage.
As you can see on the chart above the metric [# events] is growing. Does it mean that we have a feature success?
The answer is — it depends.
In fact [# events] = [# users] * [# events per user].
In some cases, we could have a situation when [# users] is rather small or stable, but [# events per user] is very high.
It could be possible when:
- a small number of power users are using a feature extremely intensively
- there is some bug in a product
In any of the above-mentioned cases the metric [# events] is growing, but the popularity of the feature — is not.
That’s why it’s better to switch from a number of events to a number of users.
Let’s check [# users].
Looking at the chart above we can prove that the metric [# users] is growing as well. Does it mean that we have a feature success?
Well, the answer is still — it depends.
Improving chat button visibility should increase [# users] of the chat feature. But our app audience is growing as well as we are doing user acquisition efforts.
That’s the reason why we need to move from absolute numbers to relatives. It means we need to assess:
- what percentage of users have started using our chat and
- whether this percentage is growing or not?
Let’s check [% users].
We see that [% users] is far beyond 1 percent and it is still growing. This is a good sign. But is this the end of the story?
The answer is — no really.
As I mentioned before we are doing user acquisition efforts. It means we are acquiring a lot of new users. And it’s possible that some groups of users could react to a product feature differently.
Let’s break down users on 2 groups: new and old.
Here I defined a new user like this: DATEDIFF( [signup_dt], [event_dt], HOUR ) < 24.
If we look at the above chart we can see that our user acquisition efforts are significant. The share of new users is 80%+.
From a feature usage analysis perspective it means that it would be better to redo our analysis for each user type separately. Such a move could give us additional insights into the adoption of the chat feature between newly acquired users and old ones.
Let’s check [% users] for new users.
There is good news here. New users are enjoying improved chat visibility, and adoption of chat is growing among them: +30% in 4 weeks.
Now let’s check [% users] for old users.
Interesting!
% of old users who are using chat is less than % of new users who are using chat. Moreover, improving chat button visibility has not impacted old users at all.
It could mean that old users get less value from a chat.
Whether to fix this for the old users or not — is an open question. For example, Netflix is always focusing on improving UX only for new users.
SUMMARY:
- Don’t use the events count, use the users' count instead.
- Look for a trend taking into account weekly seasonality.
- Always analyze a feature's popularity separately for the new users and old ones.
Next time we will try to answer the following question — how regular are users using this feature per week?
