Quick Resource Guide to UX Metrics
Product teams in many organizations want to make sure that they are delivering the right features and results. However, when you ask questions like:
- How do you know if your product is heading in the right direction?
- How do you measure progress?
- What metrics are you using?
The answers you get from product teams vary in range dramatically. On one extreme, you may hear teams say “we are drowning in data, we cannot make sense of it, we don't have the time to comb through it.” On the other side of the spectrum, teams will recognize “we don't know, we are not tracking any data.”
Here is a curated reference guide to some of the best articles available on UX metrics.
How do you set metrics? — Julie Zhuo
Zhou provides designers with a practical way to get teams working together on defining meaninful metrics.
Highlights:
- Designers who take the time to help define and understand their team’s metrics are far better equipped to drive impact than designers who aren’t thinking about them.
- The best way to select metrics isn’t to start with numbers, but rather to start with a plain-language statement about what a successful outcome would look like in human terms. In other words, how will people’s lives be improved if your efforts are successful?
- Select a single (or very small number) metric with a time dimension to measure it
- Reinforce the message by starting design presentation with the goal (the metric in plain language)
Some metrics rules of thumb
- Don’t drown in data, with a long list of metrics
- Consider counter metrics to ensure that the team is not opening a hole in one side of the experience by pursuing a metric improvement in another.
- Avoid vanity metrics (more from Jeff Gothelf later)
- Dont define metrics after the fact
Metrics vs. Experience — Julie Zhuo
It is not about one versus the other — Zhuo asks designers not to frame stuff as “Metrics Versus Experience.” She adds that metrics is not the villain.
- Evidence is key — did the change have an impact on how people use the product?
- Metrics are so valuable because they help rally a team around something clear and tangible that they can hold themselves accountable to.
- Bad things have been done in the name of “improving metrics.”
- Qualitative UX Research helps the team understand things that are hard to measure.
- Metrics “fail” where the team needs to understand issues like the cost complexity or the power of the brand.
Zhuo recommends good metrics hygene.
- Figure out which metrics are truly important, and focus on those.
- View data skeptically by suggesting counter-metrics. Every success metric should have a counter-metric
- Metrics tell you the answer to the “what” question, and designers need to ask the “why” questions to get an understanding through user research.
The Agony and Ecstasy of Building with Data — Julie Zhuo
Data and A/B test are valuable allies, and they help us understand and grow and optimize, but “they’re not a replacement for clear-headed, strong decision-making.”
Zhuo shares a contrasting set of pitfalls to watch for with each approach. Here are the pitfall lists, she provides great details in the article.
Data Pitfalls
- Picking the wrong metric to optimize for
- Over-pivoting towards what is measurable
- Biasing towards the short-term
A/B Test Pitfalls
- Spending too long perfecting tests
- Shipping successful tests right away
- Running too many tests on details that don’t matter
- Relying on A/B tests to do anything innovative or large or multi-faceted
Vanity Metrics Need to Die — UX Booth by Kristina Bjoran
- Everyone wants metrics, EVERYONE (Developers, Marketeers, Investors)
- At the heart of many product design meetings beats the drum of metrics.
- If you’re using your data only to convince someone in a position of power, you are not failing your users and yourself.
- Team manipulate data and can hide “certain truths”
Disclaimer — this article makes reference and recommends the SaaS app NomNom
10 Vanity Metrics You’re Likely Measuring That Don’t Matter — Jeff Gothelf
Gothelf describes the vanity metrics as “ often focused on product features, acquisition and adoption but do not tell us anything about product quality or meeting the user experience needs.” Some Vanity Metrics examples include:
- # of experiments
- # of customer interviews
- % of the orgnizations with continuous delivery and integration
- % of times a feature is killed
- # of engineers / or / # of UX resources
- # of usability tests
- # of times the team has pivoted
- # of people hired
- Frequency of code deployed to production
- Velocity — a) Velocity of Delivery; b) How much discovery activities
Monitoring User Experience Through Product Usage Metrics — Boxes and Arrows by Jerrod Larson and Daan Lindhout
- Usage metrics should not be considered in isolation; instead, they should be considered the starting point for additional research
- UX Teams rely on data sources and methods that could be time-consuming to create a recurring channel of in-depth UX insights
- Product Managers rely on metrics that “require little effort: to gather and report on. These metrics speak to overall quality of the business but not the specific user experience.
- Product data is strictly on user behavior, it does not illuminate user intent, expectations, or satisfaction.
- Metrics should be based on / aligned on specific product goals
- Product teams should determine and understand what metrics may be useful to them based on the nature of the app/service and business goals.
- Metrics should be tracked overtime and reviewed alongside of business metrics
Larson and Lindhout provide a sample list of metrics grouped by difference themes:
- Interaction behavior like feature use and abandonment
- Task support like completion or time
- Customer support/help like help desk tickets or calls
- Engagement like sign-ups and session duration
- Voice of customer like customer sentiment
- Client-side technology like display resolution and devices
How to Choose the Right UX Metrics for Your Product | Telepathy Kerry Rodden GV
Basic traffic metrics (like overall page views or number of unique users) are easy to track and give a good baseline on how your site is doing, but they are often not very useful for evaluating the impact of UX changes.
This is because they are very general, and usually don’t relate directly to either the quality of the user experience or the goals of your project — it’s hard to make them actionable.
Google Venture team thinks of large-scale data analysis as just another UX research method. They developed two methods to help choose and define appropriate metrics to reflect:
- The quality of user experience (the HEART framework)
- The goals of your product or project (the Goals-Signals-Metrics process)
HEART Framework
If you want your product’s design to be informed by large-scale data, it really helps to have metrics that reflect the quality of the user experience, and that map closely to your main goals.
Data-driven vs. data-informed design in enterprise products Alastair Simpson
It is not about being “data-driven,” it is about being “data-informed.” Being data-informed combines the use of Data (Quantitative — what happened) with Empathy (Qualitative — why is that happening) and Design Intuition / Execution (how we deliver and execute that experience).
DATA — provides the ‘WHAT’
Data alone is useless, maybe interesting.
EMPATHY — provides the ‘WHY’
Empathy alone is useless without meaningful insights.
INTUITION — provides the ‘HOW’
No amount of data or empathy will remove the fact that you need to essentially make decisions on how to interpret those customer insights.
[Data and Empathy] are not a replacement for clear-headed, strong decision-making. Don’t become dependent on their allure. Sometimes, a little instinct goes a long way. — Julie Zhuo
Good product design comes from striking the right balance between data, empathy and intuition.
Six Myths about Data-Driven Design — Pamela Pavliscak
Deciding how to define data is difficult for teams with spotty access to data within their organizations, uneven understanding, and little shared language.
- Data Means Numbers: The data we collect indicates what happened, but it does not answer the why question unless supplemented with qualitative methods.
- Data Is the Objective Truth: automated data collection may seem lie hard fact. Data is analyzed by humans with inherited bias. Good data describes those biases, and always provides context.
- Bigger Is Always Better: ‘Big Data’ does reveal or predict people‘s behaviors. We need meaningful categories of metrics to evaluate, understand and keep track of actionable outcomes.
- Data Is for Managers/ Developers/ Data Scientists, Not Designers: Data is not “us vs, them.” Data tells the story of the real people using technology.
- Data Kills Innovation: Data is (a) backwards looking (rear view mirror perspective; (b) tactical rather than strategic; and c) data analytics skim the surface. Data helps designers inform decisions, not drive them.
- There Is a Right Way to Use Data to Inform Design: there is not a single formula to work with data. Teams and organizations have to find their own approach in a manner that makes sense to them.
Pavliscak says that designing with data has to go beyond algorithms, automation, A/B testing, and analytics. Rather, the goal is to use all the data to develop a better understanding of everyday experience.
A big list of UX KPIs and Metrics Mr Joe Leech
If you need some ideas about metrics that may be applicable for your product, take a look as this BIG LIST compiled by Joe Leech — it includes 155+ metrics.
Key takeaways
Designers need to help their team focus on the most important and meaningful metrics. The fact that technology easily enables team to collect certain data does not make it right to spend time analyzing it. Pick a couple of metrics to determine what is happening and talk to the people who use the product to find out why.
- Pick a small number of meaningful metrics
- Use counter metrics
- Avoid vanity metrics
- Describe the metrics in plain language
- Align the metrics with the product goals
- Metrics answer the “what” and user research helps answering the “why”
- Data informs the decision maker, data does not drive the decision
- Metrics that are easy to track are not necessarily the ones that the team should concentrate on
I am sure there are many more great resources to find ideas to identify and define the perfect metrics for your product. It would be great to add more, so share your perspective.
I help companies drive results by developing and implementing UX, innovation and transformation programs.
Follow me on Twitter @jcoronado1
