Engagement Series
Why does Product Analytics go beyond product feature analysis?
Over time, users begin to use other product features, and the central value of the product is shifting.
In the previous post, I detailed to you how to think about key product events.
For each date, I took the list of product events and calculated the MCC coefficient to figure out which events were the most important in terms of returning to the product the next day.
It gave me a sense of what are a few events that were key for that day.
While this calendar-based analysis is useful, it does not unlock the full potential of this approach as at each period we have a mix of new and old users.
To overcome this drawback, I am going to switch from a calendar-based analysis to a cohort-based one.
I will take users who used the specific product features on Day 0 and then check what product features they used on Day X and how important those product features were in terms of retention on Day X+1.
For checkpoints, I will use the following days: 1st, 2nd, and 7th day.
The most important product features for each checkpoint day are at the top (they have the largest MCC coefficients).
As you can see, with time users get into the product more deeply, and their product understanding and usage are shifting.
To make it easy to understand I prepared for you a summary table with the top 5 key events on specific days.
There are a lot of insights that we can learn from the summary table above.
Key events shift
As I mentioned in the previous post key events are important signals but you need to use them wisely. They cannot be deeply analyzed in isolation from their usage context.
The product usage context includes:
- user life cycle
- the previous usage context
- the following usage context
That’s why you need to use the cohort approach here. A cohort view takes into user lifecycle in the product and shows the story of how user behavior is evolving within the product with time.
Example 1.
Product feature32 and feature28 are designed to set up and get push notifications.
It makes the most sense to expose them to users at the beginning as all the following collaboration and communication product features will rely on push notifications to deliver important information to users.
Example 2.
Product feature3, feature10/feature33, and feature17 are product features that are designed to share content, follow other users, and invite some of them for future collaboration.
Without content, adjusted virtual space, and turned-on notifications product value will be very limited. So these product features are prerequisites to unlock the next layer of product value: collaboration and communication.
Example 3.
Product feature35 is a product feature related to content search.
Users started using search only in a week when there was enough created content and finding a specific piece of content via search would be more convenient.
As we can see at different stages users hire different product features to complete specific tasks. It’s not a random walk. Users are combining different product features to get value from the product and usage context is very important here (from where, to where, and why).
Core product features
A core product feature is a product feature that is used regularly and brings steady value to users.
From the table above we can learn that the only key product feature that was regularly used during the 1st week was the product feature43. This core product feature is related to content consumption.
Is this the only core product feature?
To answer this question we need to figure out what’s the product feature usage interval:
- if a product feature is used on 4+ days in a week — it’s daily core product feature
- if a product feature is used for 3+ weeks in a month — it’s a weekly core product feature
Let me show the following example.
From the table above we can learn that product feature49 is also the core product feature. The only difference is that it’s likely used with a different tempo:
- feature43 is a daily used product feature
- feature49 is a weekly used product feature
Core feature lifetime
Probably some of the most attentive readers noticed that the core product feature43 changed a sign and started negatively impacting user retention on Day 14.
How is it possible?
Well, when users have created content, invited other users, and started collaborating around that content switching their focus to new content is already counterproductive.
That’s what distinguishes the best products from the rest. The best products focus users on achieving their goals, instead of diffusing their attention.
Zoom out
In the previous charts, we zoomed in and looked at product features on specific days. While it’s a very useful analysis, running the opposite analysis is also important. This protects you from jumping to hasty conclusions.
Let’s look at the whole table with MCC coefficients for all 15 lifetime days since signing up.
Let me show you a few examples of zoom-out analysis.
Example 1.
feature32 is a product feature related to push notifications.
From the table above we can learn that:
- starting from 2nd week since sign-up, push notifications hurt user retention. The most likely reason is that users are overwhelmed with notifications from communication and collaboration and get lost. We need to invest in prioritizing what we want to focus users on.
- there is some weekly pattern of when push notifications bring a positive impact to users (on the 6th and 13th days). It looks like some weekly push notifications constantly bring value to users. We need to figure out what they are.
Example 2.
feature17 is a product feature related to user invitations.
From the table above we can learn that:
- users continued inviting users quite regularly for the whole 2 weeks. It means that inviting users is a process that is not quickly saturated. We need to take into account this pattern and enable users to do it at any time easily.
Example 3.
feature27, feature34 and feature49 are collaborating product features.
From the table above we can learn that:
- it was the wrong assumption that product feature49 has a weekly usage interval. It’s a product feature with a daily usage interval.
- those product features become daily core features starting from 2nd week. The key moment here is that those product features are used when the product setup is complete.
Summary
Analyzing a feature’s usage is like picking up only one piece of a puzzle. You can describe it, but it probably won’t be helpful. Only when you put all the pieces together you begin to see the full picture of when and how the product delivers value to users.
To get the full product usage picture, you need to:
- use a cohort-based approach that enables you to learn how users evolve within the product
- take into account usage context
- put together zoom-in and zoom-out approaches to make strong usage hypotheses
