I recently contributed to an ongoing series called Ask Women in Product by answering this question:
I’m at a startup and I’ve been asked to define the metrics strategy for our product. What questions should I be asking to define our metrics strategy and what tips can you share so I avoid the common pitfalls?
Below is my answer in full, hope it helps!
A Metrics Strategy should start with the following components: a definition of success, a way to instrument your product, and a rubric for evaluating the product’s performance. I describe each step below so you can define a strategy from scratch, and close with a few pitfalls to avoid.
Use Your Mission to Define Success
Use your company’s mission to guide the Why of what you’re doing, and use that guidance to inform What you’re trying to achieve. A clear understanding of the Why simplifies the task of defining success when you look at the behaviors and actions that can be tracked.
For example, if your company’s mission is to create a world with no stray dogs, then your key action is placing dogs in their forever homes. You can also try this sentence structure: We want to increase the adoption of (feature) to/by (x rate) because we know that users who adopt this are (y times) more likely to (receive z benefit that ties to our mission). This statement turns your key actions into a key performance indicator (KPI) that is specific, measurable, attainable, realistic, and timely (SMART).
Instrument for a Clear View
Instrumenting, monitoring, and measuring your product’s KPIs is critical to your metrics strategy. To gauge your success in achieving the example mission to create a world without stray dogs, you want to have a measurable event for a pet’s adoption. There could be a few key actions (or data about the actions) to measure — such as the start and completion of the adoption process or the total number of rescue dogs in a facility. Invest the time to instrument products, clearly defining what will be measured and how it will be tested, and create a common language around your KPIs that teams will use cross-functionally.
A share-out of product metrics is always useful, so make your metrics available to team members and stakeholders. Steer clear of lengthy emails or posts in Slack that individuals might miss. Instead, create standardized reports that you can point to during stand-ups, or use a single slide for your all-hands or any product review meeting. The key is to provide a venue for people to ask questions and stay informed.
Regular share-outs of any product output — whether it’s research, wireframes, or metrics — will help you leverage the perspectives and creativity of your full team by creating opportunities for people to react to things in their own way.
With a common language for what success means and with definitions for your key actions, you are likely to see trends in the data. Avoid the temptation to define success based on usage trends that have already surfaced. You can resist this temptation by creating your rubric, or your lens, to evaluate this information up front.
Your rubric should consist of a set of rates or aggregate values that are the closest representation of the means to achieve your mission. Ask questions like: ”Does it matter how many users do this or how many times a day users do this? Is it critical to our success that we have a certain critical mass of users adopt this feature? Or is a steady rate of adoption more important in the long term? Why is an ‘active’ cohort defined this way?” After you answer these questions, you can document a clear set of metrics that you will evaluate consistently before, during, and after product launches.
Avoid these Pitfalls
In this section, I share some common pitfalls that I’ve encountered — and that you should make an effort to avoid — as you define your metrics.
- Shortsighted or misaligned metrics. To run with the mission to create a world without stray dogs, you might not want to base your success on the number of adoption agreements signed if several owners come back and are not able to keep their new pet for the long-term. Your data may show that three months is the key inflection point that indicates a new pet owner is fully committed to keeping their new pet, so counting the number of adoptions that reach the three-month mark is a far more useful success metric than just the number of adoption agreements signed.
- Dishonest or narrow metrics. I refer to these as “vanity metrics” because they make your product look good at a glance, but the metrics don’t reflect the actual success of your users or of the product. Examples of vanity metrics include cumulative metrics or rates that don’t make sense (e.g., top-level growth without context on marketing spend, or total logins per month when frequent, daily engagement per user is what drives your business). People rarely set out to create dishonest metrics intentionally. They typically come about when you get comfortable and depend on a never-ending stream of good news. Avoid this trap by finding and using the ratios that align with your primary product use cases and balancing these with a holistic view of your platform.
- Stale metrics. As your product develops and your onboarding techniques, interactions, or even business model changes, you’ll want to stop and recalibrate. For example, as the product matures, the reliability of the service and retention of users may become a steering force for the business, and replace the original KPIs for growth and acquisition.
After you’ve gone through the exercise of aligning your metrics around the mission, instrumenting your platform, and evaluating the product’s performance, you should have an idea of where to begin with feature improvements.
At the end of the day, if your product isn’t serving the needs of its users, then it isn’t achieving its purpose. Your product will change, and sometimes this means you’ll need to iterate on your metrics strategy as well. The intuition you build by creating a metrics strategy is a muscle that can help you make day-to-day decisions, so embrace the iterations. Good luck — I hope this helps you create a strong, defensible strategy to steer your team!
Many thanks to Rachel Bodnar for editing this piece.
This article was originally published in the August 21, 2018 edition of Ask Women in Product. Your feedback is welcome in the comments or via Twitter!