Do They Use It?
Published in

Do They Use It?

Most of the users Most of the time

Several years ago I’ve learned from Intercom blog post about Features Audit. I was impressed by importance of gathering of such data for Product Management for making Data-driven decisions.

For example, if you could have known which features are used by Most of the users Most of the time (dark green cells on image below), you could:

  • prioritize improvements for those features;
Image credit: www.intercom.com/blog/before-you-plan-your-product-roadmap

Marty Cagan, Jared Spool, Des Traynor and many many others say, that when we make Feature Audit, then we find, that features in our product are never used this way:

Image credit: www.intercom.com/blog/before-you-plan-your-product-roadmap

but always kind of this way:

Image credit: www.intercom.com/blog/before-you-plan-your-product-roadmap

It’s critical to know such Features Usage, Features Audit information about our products, right? And everyone replies “yes”. But at the same time they:

  • do not have such audit yet

I personally never seen any team who take this seriously enough to make such audits for their products for real.

Why? Because they have their Product Roadmap and it has no Analytics there. It has features to build and bugs to fix. Analytics — is always a big and difficult side-project. It is always out of scope.

From Marty Cagan interview I’ve learned that in the whole world only about 10–15% of teams are Empowered Product Teams and only these teams naturally use these techniques of Growth Hacking by applying data from Features Usage Audit. Others simple don’t.

Existing Solutions

When I start talking on this topic to IT people, they hare to name me at least “one of existing solutions for that”:

  • Google Analytics

and others. But none of these products actually solves our problem of finding “Most of the users Most of the time” features. Actually, they are all around Product Marketing, not around Product Management.

Take a look on screenshots of these products:

Google Analytics
Mixpanel
Matomo
Omniture
Heap
Adobe Analytics

They are pretty similar:

  • They are around conversions tracking;

That’s why those 10–15% of Empowered Product Teams teams develop Features Usage tools by themselves to track what matters to them. And they analyze the data by themselves: from application logs of any kind.

Solution

When I have a product look at any element at a complex interface of:

  • Web app UI

I want to know answers on 3 simple questions:

  1. Do they use it? (a button, for example)

Legit? Looks yes. So, tell me: how on Earth can I get answers on these 3 simple questions from “existing solutions” mentioned above? No way! Or very difficult and not actually what we wanted.

But take a look at this chart from my app:

Chart to answer: how many users how often use which feature and how much?

This chart is not that complex as may seem at a glance! Let me explain. Chart represents emulated data sent by some “Instagram” app:

{
"date_time": 1665864924682,
"user_id": 30,
"feature_path": ["Stories"],
}

We know, that “Instagram” app has “Posts”, “Stories” and “Reels”. And we want to know the answers on questions:

  • “do they use” “Posts”?

And if they do, then we want to know “how they use them”. And it is clearly seen on the chart!

Take a look: “pink” circles are “Stories” posts. And we can see, that:

  • 73% of unique users post “Stories” once in 3 days, average;

Totally 105% (5% — is a small mistake due to lots of math rounds while calculation of the report). Sometimes it is 98%, sometimes it is 103%, but anyway, with a small mistake now we have a real picture that answers 2 of our questions:

  • “do they (actually) use it?” and

We see exactly the picture for “Most of the users Most of the time”:

Image credit: www.intercom.com/blog/before-you-plan-your-product-roadmap
Chart to answer: how many users how often use which feature and how much?

But we have a 3-rd question: “who are they”? No problem: we have full set of User IDs (or their “salted hashes”, no problem):

and can filter data by:

  • specifying any UID or list of UIDs to display reports for them;
Report filtered for UIDs: 142 and 37 only
Report filtered for UIDs: 142 and 37 only

But this is not the end, but only beginning!

“Feature path” in a report is a list. That means that we can go deeper and analyze trees of features.

For example, “Instagram” “Stories” feature has features inside: story might be created either from “Video” or “Picture” or in “Creative mode”. And if we talk about “Video” then it could be either “Uploaded” or “Recorded”.

So, that’s how we build a Features Tree for our product:

"Instagram" -- "Stories" -- "Video" -- "Uploaded" -- ...
| | |- "Recorded" -- ...
| |
| |- "Pic" ---- "Uploaded" -- ...
| | |- "Shot" ------ ...
| |
| |- "Create" -- ...
|
|- "Posts" ---- "Photo" -- "Uploaded" -- ...
| | |- "Shot" ------ ...
| |
| |- "Video" -- "Uploaded" -- ...
| | |- "Recorded" -- ...
| |
| |- "Mix" -- ...
|
|- "Reels" -- ...
...

and now our app sends in report not just [“Stories”], but whole branch for the tracked feature:

{
"date_time": 1665864924682,
"user_id": 30,
"feature_path": ["Stories", "Video", "Recorded"],
}

And this gives us ability to navigate along features branches to continue answering all those 3 simple questions, but now for every tracked element:

Report for “Stories”: now the same chart, but for “Videos”, “Pics” and “Create mode”

I am going to continue to tell here about this Product Intelligence tool development. So, if you like the post please feel free to clap to let me know, and please subscribe to get notified about new posts.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alexey Himself

I write about practical and effective techniques that help me and my colleagues in everyday software development and testing.