Day 97 —PM series 3/5: “Outcome & Metrics”
Whether it is filling your year-end review about what business result or positive impact you create, or even the NBA Finals’ scoreboard, we’re always looking at some kind of “Outcome”. In order to understand what contributes to the outcome, we’re also looking at some types of “Metrics” (e.g. points per game for NBA players). To effectively evaluate our approach to ensure long-term product success, what kind of metrics should we measure & track?
This is a 5-part series of my work experience/knowledge about product management, please check out the rest of the series:
In today’s article, I’m going to share my experience & knowledge about “Outcome & Metrics” in the following structure:
- Value of measuring outcome for product management
- What Outcome & Metrics we should focus
- Common pitfalls
- How to effectively measure & track the metrics
Value of defining outcome and measuring
It might be common sense that we need to define desirable outcome, and use data to build better product. But, why exactly? What are the reasons behind it? Let’s start with the metrics part:
Why gathering product metrics
How do we know we’re actually making progress/ improvement? Business management guru Peter Drucker is often quoted: “If you can’t measure it, you can’t improve it.” This is true not only for business management but also for product management. In order to actually know how much progress we made, we need quantifiable evidence. By measuring specific product metrics, we can do a compare and contrast to know “where we were, where we are now”, and we can use the information to help determine “where we want to be, and what needs to be done”.
The “unknown-unknown”
Another aspect is that “we don’t know what we don’t know.” Without capturing a set of data about our product, we might assume that our design and build is awesome, but the reality might say differently. Without capturing product metrics, it could be hard to determine the WWW: “what went well” and “what went wrong”. From my experience, often times we find unexpected user pain point through identifying the anomaly in the data we capture.
For example, in the user testing, participants find our product easy to use; However when we launch the product to the market, from the metrics we found that there’s a certain step that lots of user drop out from the task/ journey. This is an important information that we didn’t expect, and that piece of info can help us for further research to understand why they drop out from the journey.
Why defining outcome
In the yesterday’s article, we talked about the importance of vision statement, which help derives specific goals we’re trying to reach. To extend that a bit, there are all types of goals: business goal, user goal, tech feasibility goal, etc. These goals might be very specific to individual teams, but they are not necessarily associate with a specific user value. For example, users might not care about “migrate from Angular to React JS”.
By defining the desirable outcome in a very specific fashion that’s associated with user behavior changes, we can help team to picture what the impact we’re trying to have with our proposed solutions. Jeff Gothelf, co-author of the book “Lean UX”, explained really well in his blog post, and you can refer to the figure below:
Let’s take that “Migrating Anguler to React JS” as an example, what’s the user value and desirable outcome we’d expect? It could be, we can lower the long term tech cost by doing so, therefore we can lower the price of the product, therefore our product has the best price-performance ratio, and therefore customers would like to choose our solution over our competitors’.
What Outcome & Metrics we should focus
How do we know what to measure to improve our product? That is a hard question for us to answer without any solid data to back us. In the modern world of product design, we shift from the guessing or “spray -n-pray” way of delivery to “evidence-based design”, which means we start with understanding the user journey and interview user before we made blind design decision.
In one of my previous articles, I wrote about “Measuring the write thing” to explain how to utilize Design Thinking and user research to define what’s important to user, what’s crucial to business, and prioritize that to determine what we want to measure. However, if you’re brand new to this data-drive product design/ development process and wonder what are the popular metrics product team would like to track, below are some common metrics track the help metrics in the following area:
User Experience
The most robust metrics framework is probably Google’s HEART, which stands for:
- Happiness: we want to know how satisfy users/customers are with our product/service;
- Engagement: how frequent do they use our product? what are the features they use? where do they stop?
- Adoption: how many new users using our product? how many people adopt the new feature we introduce?
- Retention: how many people stop using our product? how many people shift from frequent users to occasional users?
- Task Success: how easy is it to use our product? where do they drop from a task? what’s the successful rate? how long does it take for them successfully finish a task?
Business Impact
In my last article, we discussed that a great product doesn’t guarantee a market success, there are many reason like product-market fit, marketing strategy, etc. If you’re focusing on the business impact a product make, below are some typical metrics you might want to track:
- Conversion rate: new users, returning users;
- Traffic scores: direct, search, referral visitors;
- Impact of visit: interaction per visit, value per visit, cost per conversion;
- End of visit: bounce rate, exit pages
Common pitfalls
Measuring product metrics is not rocket science, but it does have some common pitfalls that we should be aware of so that we can avoid those unnecessary trial-n-error. Some examples here:
Measuring the wrong thing
This is the most common mistake in product design & development. When we forget to tie the metrics to the user journey, and capture the trivial data instead of what’s important to users, we get low-value data and doesn’t really help improve the product quality. A famous example is Marissa Mayer, Google’s former vice president of search products and user experience, decided to test 41 gradations between the two blues on a toolbar to see which got the most clicks.
Any savvy user experience designer or psychologist can tell you that there are way more other reasons that contribute to the decision of clicking on the toolbar, and the shades of blue is probably one of the least important factor there (unless it is a illegible shade from an accessibility perspective).
Interpret the metrics incorrectly
There are so many ways that people tend to interpret the metrics incorrectly. For example, falsely treat correlation as causation (e.g. in the Summer, people drink a lot of coke, in the Summer, there’s more car accident; conclusion: drinking coke causes car accident?), ignore/over-emphasize anomaly, use a snapshot of data instead of compare & contrast throughout time period, not having a good enough sample size, data or the gathering process is not kosher, etc.
Campbell’s Law
If the metrics is related to performance (e.g. by Q4 we want to increase our X rate by Y percent), the potential downside is a phenomenon that’s indicated in Campbell’s Law, which states:
“The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”
For example, if we ask the team to only focus on a small set of performance related metrics, what often happens is that teams can be focus too much on the specific improvement and get blindsided, and ignore the big picture. In other words, saves a little (on that metric) but lose a lot (in general customer satisfaction).
How to effectively measure & track the metrics
Metrics is powerful, but it’s rather a double-edge sword that might hurt by driving you into the wrong product direction. Therefore, in order to use it wisely, here are some simple steps we can follow:
#1: Define our desirable outcomes
This is THE most important step that sometimes teams missed when adopting data analytics. We want to measure we don’t blindly use data analytics just because everyone else is doing, or simply because we’re told to do so. The no.1 step is to define what a set of desirable outcomes is, so that we can choose the appropriate metrics to measure.
For example, if we’re launching a new feature on an existing product, our desirable outcome(s) could be “lots of people discover this new feature, adopt it, find it easy to use, love it, advocate for this new feature”. Features like “pull down to refresh” in your mobile apps like Gmail, Facebook are somewhat hidden to new users, and they are in need to understand how many people discover it and are actually using it.
#2: Define what we want to measure
Take that example from above, if one of our desirable outcomes is to make sure users can find our new features quickly, then what we want to measure is the findability, and the metrics we want to choose should be like task successful rate, time on task, etc.; In this case, measuing conversion rate or traffic scores is not quite relevant to our desirable outcome.
#3: Find proper tools & methods:
If we continue with the example above, it might seem obvious that writing a big survey of all different questions might not be the most effective way to help us gather meaningful data. Why? Because usually survey and other preference-gathering method won’t be able to provide you with in-depth knowledge about what actually happened, what the root cause is, and what users were doing/thinking when conducting the task (in this case, pull down to refresh the screen). Also there might be a lot of contextual factors that might impact the task successful rate, e.g. low familiarity with the app, getting distracted by the content, design, even by the contextual factors.
In situation like this, usually there are two approach that compliment each other well in order to help us get the insights we want. To go broad, we can start with general data analytics with code, or with integration with 3rd party software (e.g. Tableau). This approach helps us understand how many people find the feature, adopt the feature and become frequent users; that might give us a rough idea of the findability of the feature.
The second step is to conduct a usability test, so that we can get UX expert’s help to understand why people drop from the task, successful rate, and time-on-task. Through UX expert’s in-depth analysis, this approach can help us clarify “where it went wrong” and “why”, so that we have evidence to help us redesign or drop the new feature.
#4: Run a pilot test
Whatever approach we decide to take, it’s important to run a pilot test to understand it’s effectiveness and cost. Some of the 3rd party can get us comprehensive data, but the result set might not be able to help us pin point a specific issue that we’re trying to understand. Same can apply to user survey, focus group, usability lab test, etc. Therefore, running a pilot test can shed some light around if the approach is valid in our use case.
#5: Examine and modeling the data
Once we gather the data form the pilot test, we’d like to understand if the data can effectively provide the insights we’re looking for. 3rd party tools usually provide the modelling service in the package of offering, sometimes they’re fairly easy to use, sometimes they might not have the type of the insights we’re looking for. In general, I like what Stephen Gates said about data:
“Data gives us authenticity, creativity gives us possibility” — Stephen Gates
Sometime data along doesn’t paint the full pictures, we’ll need to imagine where it could go wrong and design our data collecting process. It’s crucial to examine the data we have, and figure out if we need follow-up testing in order to clarify the remaining questions we have before we jump into conclusion.
#6: Refine the measuring process
Now we have all the data and the insights, and we can make a sound judge of whether we’re confident with the data-collecting process or not. If we feel like we didn’t get enough data, or the data is not clean, we might want to change a tool or method; if we feel like the data results vary and not conclusive, we might want to look into how the data is collected, and consider if we need a usability test to verify where the pain points are.
Conclusion
- Before selecting your metrics you want to measure, define your desirable outcomes so that you know what to measure;
- Make sure you have a great sense of how to clean the data and enough evidence before you interpret the data;
- It’s usually more helpful to pair with quantitative research with qualitative research (e.g. usability test) to figure out the root cause, so that we don’t jump into conclusion.
Have you tried using metrics to help making product decisions? What are some challenges or pitfalls we should be aware of? I’m eager to learn from you.
ABC. Always be clappin’.
The opinions expressed in this article are those of the author. They do not represent current or previous client or employer views.
