How do you set metrics?
This was first published on my mailing list The Looking Glass. Every week, I answer a reader’s question.
How do you set metrics?
Q: What role should metrics play for designers? How should you set metrics, and how should you use them?
I’ve often heard the misconception that metrics are the domain of Product Managers and Data Scientists, cold, hard numbers used to track the business that can’t truly measure the human experience of using a product. As a result, I’ve heard of some designers being wary of metrics as a tool, believing that “data driven” development is akin to short term thinking and mindless micro-optimizations, like changing the size and color of buttons, rather than empowering designers to envision a great product.
I don’t think this is a healthy perspective, and designers who take the time to help define and understand their team’s metrics are far better equipped to drive impact than designers who aren’t thinking about them.
At a high level, metrics are simply quantifiable measures of how you’re doing. This is a good thing for clarity and tangibility — everyone on a team knows where they stand, and what “good” is. When people complain about being too “metrics-driven,” I think the root cause is when too much emphasis is placed on the value of the numbers. Simply “moving the needle on growth or engagement numbers” is pretty inward-facing and not terribly inspirational. It can feel like you’re focused on the value you’re receiving as a company, rather than the value you’re creating for others.
That’s why I’ve found the best way to select metrics isn’t to start with numbers, but rather to start with a plain-language statement about what a successful outcome would look like in human terms. In other words, how will people’s lives be improved if your efforts are successful? This should be an aspirational statement that captures the essence of the change you’re trying to make in the world. Once you’re in agreement on this statement, select a single metric (or small group of metrics) and a timeframe for measuring whether you’re on track towards achieving your goal. This won’t be perfect, but boiling the statement down to something quantifiable lets teams operationalize against it.
At Facebook, we recently used this process when updating our mission statement to focus not just on connecting the world, but also bringing the world closer together. We started with the plain-language statement where, if our efforts are successful, everyone in the world would find and participate in communities that were meaningful to them, both online and in the physical world. We then set the goal to help 1 billion people join “meaningful groups” — groups on Facebook they interact with frequently. By starting with a statement, and then using that to select our success metrics, we have a better picture in mind of the people we’re serving, and a better framework for our teams to think about when it comes to building tools to help people start and grow communities.
Note that we could have selected other metrics to look at — like maybe number of posts and comments in groups, or how many groups someone is a part of. There are different ways to approach translating your mission statement into a metric, and it’s healthy to re-evaluate periodically whether your chosen metric maps well to your actual mission. When your mission and your metrics aren’t aligned, that’s a recipe for tension, as the numbers may compel you to do things that might technically match how you’ve defined “success” but aren’t getting at what actually matters.
Now that you have metrics, how will you use them? Metrics provide clarity of focus, and should serve as the answer for “why” behind each thing you do. They let you set goals on a monthly or yearly basis, and support setting priorities and making trade-offs. When resources are constrained, the direction that’s more likely to impact your metric should take precedence over other work, and will become a useful lens through which to view every stage in your process, including design critiques, user research, and product exploration. And because the metric was derived from your mission statement, you should always be able to tie improvements to the metric to the overarching goal. It will serve you well to begin your design presentations with your goal (stated in people terms), how you measure progress against that goal, and how your work leads to a better outcome against that goal.
A few other rules of thumb with thinking about metrics:
Avoid having a huge laundry list of metrics. You can measure everything you’d like, and you should. The more you understand what people are actually doing in your product, the better you’ll be able to create the right experience. However, when setting a top-level goal to drive your team’s work, you should ideally operate around a single key metric (or at the most, a small handful) that best tracks your mission. The more metric goals you have, the more complicated it is to keep each one in mind and make trade-offs against them. While it may feel painful, making this explicit decision up front can save you many headaches of later of arguing about what work is more important.
Consider tracking counter-metrics. There are certainly many caveats to the above idea of setting a goal around a single metric. A notable one is that you may need to have a counter-metric to help balance short term and long-term trade-offs. As an example, if your metric is engagement, as measured by “increasing time spent per user”, you may want to include the counter-metric that you will not decrease the number of engaged users. The goal is still the same one primary metric (time spent per user), but the counter-metric allows you to keep in mind that systems are complex, and keep you from over-optimizing for a subset of your audience at the expense of creating more broadly appealing experiences.
Avoid vanity metrics. Much digital ink has been spilled on this topic, so I’ll keep this brief: Unless a metric truly captures the value you’re creating for people in an ongoing way, it’s not the right metric. Vanity metrics that sound fluffy and fun (like “number of registered users”, which is often much bigger than daily actives) can actually do more harm than good, as they can lull you into thinking you’re doing a good job when all you’re doing in churning through your audience. Metrics that are less likely to be in vain are retention, revenue, daily usage, and meaningful actions.
Avoid subjective measures. As much as possible, try to understand how you’ll quantify the value you’re creating. Sure, there are things you’d love to achieve, like “people love our product”, but if you’re trying to measure sentiment, consider net promoter score or other ways to understand numerically where you currently stand (or how you fare relative to the market), and set a goal for the change you’re hoping to make in a way that will be measurable. Looking at a few anecdotal stories or customer e-mails is not a rigorous way to draw conclusions if your actual user base is diverse.
Avoiding setting metrics after the fact. Trying to decide whether or not you were successful after your product is already out in the market doesn’t make much sense. We humans will tend to see the most optimistic version of the facts. The whole idea of data informed development is that you want to avoid being biased. You should know, going into your work, what your goal is, and how you will measure if you achieved it.
Consider setting 50th percentile goals. Your goals should ideally stretch your team. You want to find the right balance between selecting a goal that is so hard that it seems unlikely to be met, or one so easy that you’ll be able to reach it by effectively doing nothing. One way to describe this is by thinking about your goals as being 50th percentile goals — i.e. there is a 50% chance you will reach your chosen metric goal. This means that if you are hitting your goals half the time, you are doing a good job. If you’re always reaching or exceeding your goals, you likely didn’t set your sights high enough. And if you don’t reach your goal, you should discuss honestly whether the goal was wrong, or if the breakdown was something in the team’s strategy or execution. In all cases, getting good at setting the right metrics is itself a skill, and will be something your team should work to improve at over time. If needed, it may be helpful to further dig into what the 90th and 10th percentile cases are as well, i.e. for your given metric, in addition to the 50p goal, what are you 90% sure you can reach (the 90p case), and what’s the wildly ambitious “if everything goes well” number that you’re only 10% confident you can hit (the 10p case).
Don’t get bogged down on semantics. What’s the difference between goals, metrics, measures, drivers, aims, etc…? Sometimes people use different words to refer to different things when discussing metrics. I’m not too worried which words you’re using, so long as you’re aligned on definitions with your team.
In the end, there’s no one ideal way for your team to select and use metrics, as it will depend heavily on the factors specific to your team, product, and market. The main message is simply that metrics should be embraced and play a critical part in the role of a product designer. This is a topic that has become quite dear to me, as I’ve seen time and time again how designers who use metrics well can be incredibly successful. If you’re interested, here are some other related articles I’ve written on this topic:
To ask a question or follow along weekly with more Q&As like this, subscribe to The Looking Glass mailing list.
