You’re Using Metrics Wrong — Here’s How Not To
We’ve all been there.
Some boss’ boss or VP wants to know how many active users we have, or what app is most popular, or least popular, or what the average time on app is for a given user segment. So being the good head’s of product or PMs we are, we go find out, put it in a dashboard, make it look pretty and ship it.
Congratulations, we’ve just managed to cross-collaboratively waste everyone’s time.
Measuring things for the sake of measuring them is like me blindly telling you how many outlets I have in my house.
Not interesting and Not useful.
I already know your response, because it was how I felt:
“Ok, know-it-all, what would you do?”
I’m glad you asked.
This is a problem that Product Managers across the industry face. No matter if you work for multi-thousand-employee FAANMG company or a 30-person startup, having to come up with Metrics That Matter; ones that align product teams and actually make meaningful impact in extremely ambiguous scenarios is one of the more difficult parts of the job.
The overarching question you have to ask is:
Why Do We Care?
Let’s dig in.
Metrics on their own are almost never useful (yeah yeah, tell me in the comments your super special exception to this), Rather PMs should ALWAYS be starting from some version/combination of the following questions:
- What customer problem(s) do I want to solve?
- What product use cases do I want to replicate?
- What product outcomes are we not achieving?
- What product outcomes do we want to stop achieving?
Once your team begins to uncover answers to these, you throw them into a product hypothesis bucket to examine closer, much like a detective.
Only then should you start deciding what data points and dashboards you need to dig into in order to determine the best path forward.
Let’s look at an example, first with a metric request and then with the Metrics that Matter approach:(Note, I don’t work for Microsoft and have never worked on the Teams product):
Customizable Top Bar in Microsoft Teams
Scenario 1: Someone high up asks “Show me the actions in teams ranked by most clicked to least clicked”
Outcome: A dashboard that shows this. No product change, no meaningful insight, nothing really actionable.
— — — — — — — — — — — —
Scenario 2: Metrics that Matter approach:
What customer problem do I want to solve?
- We have lots of user segments with lots of different teams use cases. We think it’s important to let users decide the actions they want available in their quick select top bar (shown below).
What product outcomes are we not achieving?
- Actions that user care about (like seeing meeting options, or record) are hidden behind the “More actions” ellipse (below the proverbial product fold), we want to give important actions relevant weight.
There are many different potential product changes that would work to address these two items, but we are focused on data. So we need to see if the data says this is actually a problem.
We can start by pulling the same data that is in scenario 1, but now instead of just seeing numbers we can mark all the actions that are behind the ellipse. We can also pivot the data based on various customer segments to see if different user types really use actions with more or less frequency. Once we have that info we can begin to put forward product experiments to validate our assumptions.
We have pulled almost the same data sets in both scenarios, but in scenario 2 because we have customer hypothesis as a starting point, the data actually means something.
In Practice
The takeaway here should be that as a product person, when anyone ever wants to look at metrics, you should ALWAYS ALWAYS be asking that ever important question: Why Do We Care? and rather than using metrics as a meaningless vanity measure, use them to actually validate or invalidate product hypothesis and make meaningful change.
Don’t measure to measure, measure for change.
Note: would love to hear your metrics experience, drop a comment below.
