Creating User Centered Metrics
Issue 20.1
Adventures in UX Design is a newsletter helping you navigate UX roadblocks
(For more on the topic of metrics for UX design, check out issue 20.2, Making Metrics Matter in UX Design.)
We gather data to determine the direction and assess the value of what we create. Every business, and client, has a specific approach to the way they define success. How many subscribers do you have? What is the customer retention rate? How many unique visits, and downloads? What is the time on site, the bounce rate? How are your WAUs, DAUs, and MAUs? (Weekly, Daily, and Monthly Average Users.) How do you define engagement?
We make critical decisions around content, design, and product development that are informed in part by metrics. But what do metrics tell us about the success or failure of a product, the user experience, or design? What metrics should we rely on?
Unfortunately, user experience designers rarely hold sway over determining what metrics matter, but they may be able to shift the conversation around how teams determine metrics for success to evaluate product designs and usage.
The quantitative data that we collect often drives important product decisions, but numbers garnered from out-of-the-box solutions like Google Analytics often generate arbitrary data points that don’t really tell us much. Metrics paint only part of the picture: the what, and not the why. For the why to happen, we need qualitative measures. We need the narrative behind the numbers to establish and track metrics that provide true insight into the user experience, the value, and the outcome of what we create.
Establish The Right Measures
When we use metrics alone to assess the value of something, we fall into the realm of making half-informed decisions based on somewhat arbitrary indicators that can hold sway over a business.
Is the high bounce rate good because the user has come and found what they needed and left? Or is it bad, because they came and left disappointed? Is time on site an indicator of a customer’s appreciation of the product and content, or a reflection of a customer’s frustration at not being able to find something (or a browser left open that timed out of the session)?
Obsessing over numbers derived from out-of-the-box analytics can pitch our process into an endless loop of trying to move numbers in a direction that can’t tell a full story, and may even lead some of us into implementing dark patterns to reach those numbers.
When the goal becomes a number, the human element, the user experience, gets lost. Gerry McGovern shares a story about Irish Police being criticized for falsifying the number of Breathalyzer tests given, because the department’s emphasis for success fell not on the number of drunk drivers stopped and prosecuted, but rather on the number of tests given.
McGovern calls this a “cult of volume” and deftly shows how metrics can drive behavior and culture. An overemphasis on outputs over outcomes perverts systems, especially metrics that are prone to manipulation and interpretation over observation, explains McGovern.
As we explore in a UIE seminar on metrics, designers and product leaders should combine qualitative and quantitative data to assess the performance of their product designs if they want to make better decisions.
We break down the quantitative language used when we talk about tracking performance:
- Measure: Something we can count
- Metrics: A measure we can track
- Analytic: A measure that software tracks
We use data derived from the observation of our products, both qualitative and quantitative, to find the right solutions. We make inferences from those observations that lead to solutions. But we cannot choose the right solution if we only have the numbers and rely on our assumptions to fill in the rest of the story.
“Google Analytics and all the analytics tools, for the most part, there’s a raft of things they can’t tell you. They cannot tell you what’s useful. They cannot tell you who is spending the most money on your site. They cannot tell you how to improve your content. They can’t even tell you why somebody clicked on something. We can collect all the observations we want, but the inferences are left open, because Google Analytics will not tell you why,” explains Jared.
Teams need better, more targeted metrics to improve the user experience. One tool that can get at the heart of the user experience is a journey map. Drawing qualitative and quantitative data together into a journey map can illustrate for teams why people are frustrated or delighted about an experience, and where opportunities lie to improve the product.
Targeted metrics applied to qualitative research results in a powerful addition to the UX designer’s toolkit.
READ: Metrics Drive Behavior and Culture by Gerry McGovern
WATCH: Is Design Metrically Opposed with Jared Spool
