Metrics Versus Experience

Julie Zhuo
Jul 6, 2016 · 10 min read
Image for post
Image for post
Photo by ChocolateFrogs
  • How can we balance driving up numbers and doing something meaningful?
  • And my personal favorite: Are you that data-driven, or do you actually care about the user experience?

First off, don’t frame stuff as “Metrics Versus Experience.”

Besides evoking big-budget movies pitting two passionate-fanbase superheroes against each other, framing things as “metrics versus experience” is entirely the wrong way to start the conversation.

Certainly, bad things have been done in the name of “improving metrics.”

Just like how you can eat too many jelly-filled donuts and give carbs a bad name, so too can metrics be used to justify poor decisions. This can happen because not everything you can measure is worth measuring or affecting. This can also happen because you don’t get the full story by looking at a single metric. Often, you need a suite of metrics to get a really good picture of what’s actually happening.

Certainly, there are important things we can’t easily or accurately measure.

If we could read user’s minds, then we could in theory design the perfect experience for them. Unfortunately, we’re not all Jean Greys, so we make due with what we can measure to try and take educated guesses as to what people care about. In this day and age, what we can measure has its limits, and it’s important to always remember that. Simply looking at what people are doing in your product can’t tell you:

  • whether a change increases or decreases people’s trust in your product over time
  • how simple and easy to use your product is perceived to be
  • how people see your product versus other similar products in the market
  • what things people most want changed, added, or fixed
  • how people will want to use your product as time passes
  1. Understanding the power of brand: when Apple or Nike comes out with a new product, lots of people are inclined to buy it, even without doing their research, because they’ve had an awesome experience with that brand in the past. The same would not be true if some new upstart called Pear or Sike came on the market with an equivalent product. At a high level, we all know and understand this. However, it’s hard to quantify the power of brand and turn it into a number that can be tracked every day. It’s hard to know how all the thousands of decisions a company makes impacts that brand, and what the costs and benefits are in weighing those tradeoffs.
  2. The power of big bets: no metric can tell you what the bold strokes needed to win the future are. Imagine 2008, when smartphones were just starting to emerge. If you looked at the metrics for your website, you would have seen a tiny sliver of traffic coming from smartphones. You may have concluded, very practically, that you shouldn’t really invest too much into building for mobile since it’s such a small part of your audience. Today, we realize the vision and foresight of those who did bet big on mobile and reaped huge rewards. No examination of current behavior can accurately tell which way you need to leap. Strategic, long-term planning still requires much of the same thing it always did: trusting your gut.

Some rules of thumb for good metrics hygiene:

These are some of my biggest learnings in my quest to become more and more disciplined about the tactics of good goal setting and measurement.

  • To optimize for growth, understand your funnel. In order for people to become regular users of your product, they have to pass through a bunch of hurdles. First, they have to be aware of your product. Second, they have to be interested enough to check it out. Third, they have to convert (download an app, fill out a form, confirm e-mail, etc.) Fourth, they have to do enough within your product to understand why it might be valuable in their lives. Fifth, they have to remember to come back. At each of these steps, you will lose people. If you can track and measure what that rate of loss is, you can then start to figure out where to focus your efforts to make your funnel less leaky.
  • Figure out which metrics are truly important, and focus on those. It’s tempting to get into the state where you track everything (because you can), and you have a dashboard filled with numbers that all feel like they should be green. Recognize that most things don’t matter, and that only a small handful actually do. Don’t waste time talking about the unimportant stuff, and don’t sweat letting some of the less important metrics go up or down.
  • To figure out the best metric to track, use the magic-wand technique. Ask yourself: “If I could wave a magic wand and know anything about my users in the world, what would I most want to know to tell me whether my app will be successful?” Even if your answer is not something you can actually measure (“Is my app suggesting recommendations that my users find valuable?”), it is a helpful starting point to work from. (“Okay… so I can’t ask every user if the recs were valuable… but if it were valuable, I’d probably see them saving or sharing recs more, and they’d probably spend more time reading recs, and…etc, etc.”)
  • Don’t just accept a metrics goal without understanding it. I can’t emphasize this enough: the goals you and your team agree to will be hugely impactful to your work, so make sure you buy into them. Do not accept metric goals at face value. Ask why. Ponder whether or not they make sense, and what behaviors they will incentivize. Are there situations where something will feel like a good decision but the metric doesn’t move? Conversely, are there situations where you could imagine the metric going up a lot but not be convinced that the product is actually better? If so, would another metric (or set of metrics) do a better job of tracking what actually matters?
  • View data skeptically by suggesting countermetrics. If the data is showing you what look like good results, ask yourself: “What else can I look at to convince me that these results aren’t as good as they seem?” These are called countermetrics, and every success metric should have some. (For example, don’t look at click-through rate without looking at the number of fast bounces back, don’t look at the sales numbers of a product without looking at how many returns or cancellations there are, etc.) It’s much better to be paranoid about interpreting data so you can quickly catch your mistakes and adjust your strategy. Don’t fall into the trap of confirmation bias where you’re just looking for signals that prove your intuitions are right.
  • Use qualitative research to get at the why. Quantitative data that tells you what people did is best paired with qualitative research that give you insight into how people felt. Conduct usability testing, utilize focus groups, and run surveys to get at the why behind the behavior you’re seeing.



The Year of the Looking Glass

A collection of essays by Julie Zhuo on design, building…

Julie Zhuo

Written by

Currently: Inspirit. Former Product Design VP @ FB. Author of The Making of a Manager https://amzn.to/2PRwCyW. Find me @joulee. I love people, words, and food.

The Year of the Looking Glass

A collection of essays by Julie Zhuo on design, building products, and observing life.

Julie Zhuo

Written by

Currently: Inspirit. Former Product Design VP @ FB. Author of The Making of a Manager https://amzn.to/2PRwCyW. Find me @joulee. I love people, words, and food.

The Year of the Looking Glass

A collection of essays by Julie Zhuo on design, building products, and observing life.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store