Most of the users Most of the time
Several years ago I’ve learned from Intercom blog post about Features Audit. I was impressed by importance of gathering of such data for Product Management for making Data-driven decisions.
For example, if you could have known which features are used by Most of the users Most of the time (dark green cells on image below), you could:
- prioritize improvements for those features;
- prioritize smoke and regression tests for those features;
- reprioritize or even remove features that less than you thought or actually never being used;
- reprioritize or even remove smoke and regression tests for those features;
Marty Cagan, Jared Spool, Des Traynor and many many others say, that when we make Feature Audit, then we find, that features in our product are never used this way:
but always kind of this way:
It’s critical to know such Features Usage, Features Audit information about our products, right? And everyone replies “yes”. But at the same time they:
- do not have such audit yet
- do not have resources and enough will to implement such analytics for their products
I personally never seen any team who take this seriously enough to make such audits for their products for real.
Why? Because they have their Product Roadmap and it has no Analytics there. It has features to build and bugs to fix. Analytics — is always a big and difficult side-project. It is always out of scope.
From Marty Cagan interview I’ve learned that in the whole world only about 10–15% of teams are Empowered Product Teams and only these teams naturally use these techniques of Growth Hacking by applying data from Features Usage Audit. Others simple don’t.
Existing Solutions
When I start talking on this topic to IT people, they hare to name me at least “one of existing solutions for that”:
- Google Analytics
- Mixpanel
- Matomo
- Omniture
- Heap
- Adobe Analytics
and others. But none of these products actually solves our problem of finding “Most of the users Most of the time” features. Actually, they are all around Product Marketing, not around Product Management.
Take a look on screenshots of these products:
They are pretty similar:
- They are around conversions tracking;
- Show “average temperature” among all users (but you might want to see the data only for trial customers or only for paid customers or only for highly paid ones);
- No any chart of that “features comparison” to find “Most of the users Most of the time”
That’s why those 10–15% of Empowered Product Teams teams develop Features Usage tools by themselves to track what matters to them. And they analyze the data by themselves: from application logs of any kind.
Solution
When I have a product look at any element at a complex interface of:
- Web app UI
- Mobile app UI
- Server app API
I want to know answers on 3 simple questions:
- Do they use it? (a button, for example)
- If they use it, then how they use it? (how often, when, how many of them, what value is set / chosen / entered most or least often, etc.)
- Who are they actually? (are they those whom I interested in)
Legit? Looks yes. So, tell me: how on Earth can I get answers on these 3 simple questions from “existing solutions” mentioned above? No way! Or very difficult and not actually what we wanted.
But take a look at this chart from my app:
This chart is not that complex as may seem at a glance! Let me explain. Chart represents emulated data sent by some “Instagram” app:
{
"date_time": 1665864924682,
"user_id": 30,
"feature_path": ["Stories"],
}We know, that “Instagram” app has “Posts”, “Stories” and “Reels”. And we want to know the answers on questions:
- “do they use” “Posts”?
- “do they use” “Stories”?
- “do they use” “Reels”?
And if they do, then we want to know “how they use them”. And it is clearly seen on the chart!
Take a look: “pink” circles are “Stories” posts. And we can see, that:
- 73% of unique users post “Stories” once in 3 days, average;
- 22% of unique users post “Stories” once in 4 days, average;
- 6% once in 2 days;
- 4% once in 5 days;
- 1% once in 6 and 7 days;
Totally 105% (5% — is a small mistake due to lots of math rounds while calculation of the report). Sometimes it is 98%, sometimes it is 103%, but anyway, with a small mistake now we have a real picture that answers 2 of our questions:
- “do they (actually) use it?” and
- “how (often, much) they use it?”
We see exactly the picture for “Most of the users Most of the time”:
But we have a 3-rd question: “who are they”? No problem: we have full set of User IDs (or their “salted hashes”, no problem):
and can filter data by:
- specifying any UID or list of UIDs to display reports for them;
- specifying any UID or list of UIDs to exclude them from reports;
But this is not the end, but only beginning!
“Feature path” in a report is a list. That means that we can go deeper and analyze trees of features.
For example, “Instagram” “Stories” feature has features inside: story might be created either from “Video” or “Picture” or in “Creative mode”. And if we talk about “Video” then it could be either “Uploaded” or “Recorded”.
So, that’s how we build a Features Tree for our product:
"Instagram" -- "Stories" -- "Video" -- "Uploaded" -- ...
| | |- "Recorded" -- ...
| |
| |- "Pic" ---- "Uploaded" -- ...
| | |- "Shot" ------ ...
| |
| |- "Create" -- ...
|
|- "Posts" ---- "Photo" -- "Uploaded" -- ...
| | |- "Shot" ------ ...
| |
| |- "Video" -- "Uploaded" -- ...
| | |- "Recorded" -- ...
| |
| |- "Mix" -- ...
|
|- "Reels" -- ...
... and now our app sends in report not just [“Stories”], but whole branch for the tracked feature:
{
"date_time": 1665864924682,
"user_id": 30,
"feature_path": ["Stories", "Video", "Recorded"],
}And this gives us ability to navigate along features branches to continue answering all those 3 simple questions, but now for every tracked element:
I am going to continue to tell here about this Product Intelligence tool development. So, if you like the post please feel free to clap to let me know, and please subscribe to get notified about new posts.
