{Product Notes} Defining a Metrics Strategy

Image for post
Image for post

I recently contributed to an ongoing series called Ask Women in Product by answering this question:

I’m at a startup and I’ve been asked to define the metrics strategy for our product. What questions should I be asking to define our metrics strategy and what tips can you share so I avoid the common pitfalls?

Below is my answer in full, hope it helps!

A Metrics Strategy should start with the following components: a definition of success, a way to instrument your product, and a rubric for evaluating the product’s performance. I describe each step below so you can define a strategy from scratch, and close with a few pitfalls to avoid.

Use Your Mission to Define Success

For example, if your company’s mission is to create a world with no stray dogs, then your key action is placing dogs in their forever homes. You can also try this sentence structure: We want to increase the adoption of (feature) to/by (x rate) because we know that users who adopt this are (y times) more likely to (receive z benefit that ties to our mission). This statement turns your key actions into a key performance indicator (KPI) that is specific, measurable, attainable, realistic, and timely (SMART).

Instrument for a Clear View

A share-out of product metrics is always useful, so make your metrics available to team members and stakeholders. Steer clear of lengthy emails or posts in Slack that individuals might miss. Instead, create standardized reports that you can point to during stand-ups, or use a single slide for your all-hands or any product review meeting. The key is to provide a venue for people to ask questions and stay informed.

Regular share-outs of any product output — whether it’s research, wireframes, or metrics — will help you leverage the perspectives and creativity of your full team by creating opportunities for people to react to things in their own way.

Evaluate Performance

Your rubric should consist of a set of rates or aggregate values that are the closest representation of the means to achieve your mission. Ask questions like: ”Does it matter how many users do this or how many times a day users do this? Is it critical to our success that we have a certain critical mass of users adopt this feature? Or is a steady rate of adoption more important in the long term? Why is an ‘active’ cohort defined this way?” After you answer these questions, you can document a clear set of metrics that you will evaluate consistently before, during, and after product launches.

Avoid these Pitfalls

  • Shortsighted or misaligned metrics. To run with the mission to create a world without stray dogs, you might not want to base your success on the number of adoption agreements signed if several owners come back and are not able to keep their new pet for the long-term. Your data may show that three months is the key inflection point that indicates a new pet owner is fully committed to keeping their new pet, so counting the number of adoptions that reach the three-month mark is a far more useful success metric than just the number of adoption agreements signed.
  • Dishonest or narrow metrics. I refer to these as “vanity metrics” because they make your product look good at a glance, but the metrics don’t reflect the actual success of your users or of the product. Examples of vanity metrics include cumulative metrics or rates that don’t make sense (e.g., top-level growth without context on marketing spend, or total logins per month when frequent, daily engagement per user is what drives your business). People rarely set out to create dishonest metrics intentionally. They typically come about when you get comfortable and depend on a never-ending stream of good news. Avoid this trap by finding and using the ratios that align with your primary product use cases and balancing these with a holistic view of your platform.
  • Stale metrics. As your product develops and your onboarding techniques, interactions, or even business model changes, you’ll want to stop and recalibrate. For example, as the product matures, the reliability of the service and retention of users may become a steering force for the business, and replace the original KPIs for growth and acquisition.

Conclusion

At the end of the day, if your product isn’t serving the needs of its users, then it isn’t achieving its purpose. Your product will change, and sometimes this means you’ll need to iterate on your metrics strategy as well. The intuition you build by creating a metrics strategy is a muscle that can help you make day-to-day decisions, so embrace the iterations. Good luck — I hope this helps you create a strong, defensible strategy to steer your team!

Many thanks to Rachel Bodnar for editing this piece.

This article was originally published in the August 21, 2018 edition of Ask Women in Product. Your feedback is welcome in the comments or via Twitter!

Written by

mars bound

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store