🚀 Measuring Product Success — Three Tiers of Product Success Metrics 📈

A lot of people and organizations talk about value. Many want to deliver more value or unlock new value for their customers, users, and business stakeholders. The same goes for product professionals. Most of us want to maximize the value of our product. A lot of talk about value. However, only a few of us are able to actually estimate, track, measure, and steer on value, in order to maximize it. One reason is that project-based development approaches still dominate at many organizations, leading people to assume that a project-management framework is the same thing as having a product-management framework. But, a project is a discrete piece of work with a deadline and specific outputs to be delivered. Projects are an essential part of product development, but thinking only in terms of projects leads many of us to the Build Trap.

Companies stuck in the build trap measure their success in terms of outputs rather than outcomes. They stop producing real value for their users, lose market share, and are vulnerable to disruption. They end up in the build trap when they associate value with the number of things they produce — outputs like products, features, and releases — instead of the outcomes they want to create for their customers. A sales-led company can end up shipping 30 features that no-one wants. A visionary-led company can be a powerful organization, but innovation needs to be part of its DNA or the vision becomes dependent on just one individual. A technology-led company is likely to lack a market-facing, value-led strategy.

Product-led companies, however, optimize for outcomes, align their product strategy to specific goals, and prioritize the projects (or Epics or Initiatives if you will) that will develop those products. They actively track and manage how their products perform, using the right Product Analytics Metrics and Value Metrics. Not all metrics are the same. And not all metrics are useful for tracking value. Therefore, in this article, you will explore the three different tiers of value metrics.

🔥 Metrics will help you better predict what will resonate with customers.

Before we jump into the Three Tiers of value metrics and measures, it’s worth taking a moment to underline a few reasons why it’s essential for any product leader to measure what matters. We would argue the most important reason to track and analyze a set of product (value) metrics is that it helps you evaluate and predict customer needs. These metrics, sometimes called key performance indicators (KPIs), give the company quantifiable evidence about which aspects of the product or customer experience are resonating with customers — and which aren’t. Product, marketing, and sales teams can all use this data-driven information to understand what motivates their customer “personas”. This data helps the company continually improve its products. Without such metrics, product managers are forced to rely on educated guesses when deciding which products or features to prioritize. As Product Director Paul Yokota explains in a podcast episode on This Is Product Management:

“A product manager’s intuition is valuable, but gut instinct should only be applied in conjunction with product metrics.”

Not all metrics are equal. The quantity of data points that you and your team measure are not as necessary as the quality of that data. In the next part, we have broken down metrics into three tiers:

Tier 3 — Vanity metrics
These are the vanity metrics that can boost a product team’s ego. You’ll want to be cautious about putting too much weight on them.

Tier 2 — Proxy metrics
The metrics in tier 2 are proxy metrics that can suggest something about your product’s potential success. But because they do not themselves represent direct evidence of how your product will resonate with users, you need to be careful about these metrics.

Tier 1 — Business and customer metrics
These are the business and customer-oriented metrics, and they’re the ones you’ll want to focus on primarily. Tier 1 metrics — sometimes called Key Performance Indicators (KPIs) — capture concrete data about things like revenue, customer retention, acquisition, the size of your user base, and more.

Let’s explore each category in more detail below.

👻 Tier 3 — Vanity Metrics

What tier 3 metrics typically look like:

• Quantity of social media followers
• Number of views of a promotional video
• Popularity of your product’s user-discussion space

Why they’re the least valuable metrics:
Let’s say your product team creates a chat area within your product online, where users can comment on the product and share thoughts, ideas, and best practices. Now imagine that this area of your product gets an enormous amount of both comments and viewer traffic.

Visitors are posting product ideas and making comments about their jobs or industry, or even their personal lives. Many of these comments start long and thoughtful threads with other visitors. Disagreements and even arguments break out. Some of your “power” commenters even start treating your product’s chat area like a mini-Medium site, posting long essays such as “10 ways to streamline your daily workflow.” After all, many of your product’s users are in similar roles, so they have a lot in common. Soon, your chat area is looking like a small-scale Twitter, with tens or even hundreds of thousands of views.

Question: Does all of this engagement tell you anything about how your product is resonating with customers or how well it will do in the coming months?

Not necessarily. What might have happened here is that people your chat area created a useful channel for your user personas to connect with like-minded people? Some of these conversations might be about your product, but many of them could also be much broader and not related to your company.

It’s great that you’ve created this accessible digital space to attract your user persona. Some of these people might become interested in your product, even if they arrived at your chat space only because they found a relevant discussion thread in a Google search that led them to your company. But, it’s important not to read too much because many people are using some aspect of your product to connect. Until this heightened attention to your chat area leads to your product’s actual usage — or better yet, purchases of your product — you need to recognize this large number of “commenters” for what it is: a vanity metric.

How a vanity product metric can go wrong

Let’s say your company makes an app that lends itself to be kept open for users’ long durations. Consider a timer app that helps consultants or freelancers track the time they work on projects. If your team uses “average user session time” as a product metric to gauge how much value users derive from your product, you’ve mainly created a “vanity metric.” It might feel great to say that the average user keeps your app running for several hours at a time. But you know this isn’t because those users are actively engaged with your product — you’ve designed it to run in the background, to be more or less ignored once your users set the initial timer.

Vanity metrics can be fun to follow. They can increase your team’s morale. They can even serve as very indirect indicators that your products have an increased potential for success. But don’t be tricked into believing a high number of likes or hearts on social media will necessarily translate into anything that moves the needle for your business.

👍 Tier 2 — Proxy Metrics

What tier 2 metrics typically look like:

• Level of user engagement and interest in your product survey
• Number of prospects who take action showing interest in your product, such as downloading a document about it
• Ideas for features ideas from prospects or customers

When you’d use them:
Sometimes you need these metrics because you don’t yet have a product on the market or even a prototype to share with users. In these cases, you might need to gauge interest levels among your target market and estimate how much success your product might enjoy once you release it.

Their benefits and shortcomings:
Tier 2 metrics are more valuable than vanity metrics because they can tell you something about your product and the target user. But they are typically capable of providing only indirect measurements of the data you want — such as the size of your market and what percentage of that market you can expect your product to capture.

For example, let’s say you send out a request to a list of prospects asking for feature ideas for a new product you’re considering. And let’s imagine you receive a large number of responses. Many of the suggestions for features are logical, thoughtful, and creative. Those ideas themselves might prove valuable if you decide to pursue this product.

But neither the number of responses nor how well thought out they are can tell you whether or not you have a viable market for the product. It can tell you only that some people found the product concept intriguing in theory. Or it might be telling you that a number of your target customers enjoy a good brainstorming session about product ideas.

You will need to do several things to distill the exact level of interest in your product from the other signals these metrics might be telling you.

For example, you’ll need to:

• Check the respondents who who submitted positive and thoughtful feedback against those who have responded to your surveys in the past. If you’re seeing the same people, consider whether they’ve purchased your products in the past when reviewing their feedback.
• Review the survey results against similar feedback your company has received about previous products, and then review how well those products performed in the market. If strong survey results have translated into product success in the past, that data point is valuable. But if you’ve experienced disappointing market results after receiving encouraging feedback from your user personas before, you can’t assume a direct connection between positive survey responses and product success.

Tier 2 metrics can be valuable. But generally speaking, you’ll want to minimize your dependence on these metrics and focus instead on the Tier 1 metrics we’re going to discuss in detail next.

Note: Although the sweet spot will differ for every company in terms of how many unique metrics to track and analyze, a good rule of thumb is to limit your count to five or six. It is vital to limit your metrics to avoid getting confused or overwhelmed by data and, at the same time, reduce the chances that you’ll end up analyzing data points that are of low value.

🤑 Tier 1: Business & Customer Metrics

Business, North Star, and customer-oriented metrics are the most useful for tracking and analyzing because they can give you direct data about the success of your products and your business. These metrics tell you directly how your product performs in the market and how you can expect it to continue functioning. This category represents the only genuine product success metrics. These are also the most valuable of all types of metrics because they’re the best indicators of how successfully your team executes its product strategy.

When you first established your product’s strategy, you answered important high-level questions such as:

• Who will our product serve?
• How will this product benefit those people?
• How will this product support our company’s goals (in terms of market share, revenue, profitability, customer loyalty, etc.)

The only direct way to quantifying your team’s success at achieving these goals is through the various Tier 1 metrics. If you had to focus on just one metric to measure your product’s success, you’d want to ensure you pick a Tier 1 metric. We will figure out in the next lessons what Tier 1 metrics are and how to measure them.

Another reason the tier 1 metrics we’re about to discuss can be invaluable is that they can create a single source of truth for your entire organization. These metrics represent hard, indisputable data about how your product performs in the market.

The right metrics mean you can give all of your stakeholders across the company (in marketing, sales, support, operations, finance, and management) an accurate picture of how the product performs.

Sharing these tier 1 data points throughout your company ensures everyone is working from the same information, which will help them make better-informed decisions.

🏁 Wrap-up

Hopefully, this article offered you some inspiration for tracking and measuring your Product’s Value in practice. There are many tools, techniques, processes, and frameworks that can be used to become a more value-focused or product-led organization.

The contents from this article are covered into much more detail in our Professional Product Management Training Module: Value Maximization. So, if you want to collaborate, get trained, facilitated or if you want some help getting your Value Dashboard, Value Estimation, Feature Prioritization, or other tools and techniques in place, please reach out! We are happy to help.

🎓 Excited to move your Product Management skills forward?

Becoming an expert in a field takes more than a single course. Consider it to be a journey, requiring knowledge gathering and experience in practice. That is why we have developed Product Management Learning Journeys for Product Owners, Product Managers, and Product Leaders.

We have found that people often want a structured approach to boost just those skills that they need to improve in a specific section. In order to enable that, we have set up various Professional Product Management Training Modules. Modules like Value Maximization, Envisioning & Storytelling, Strategy & Roadmapping, or Stakeholder Engagement & Politics provide you with 8+ week learning and development tracks, allowing you to really improve a specific area of competence, instead of going through a very generic course that covers all kinds of different topics on a high-abstraction level.

A typical Professional Product Management upskilling track at Xebia Academy

Our learning journeys are designed to find the perfect balance between the theory from university with the intensity of a bootcamp. These ingredients are blended into a training format that fits anyone’s preferred style of learning. We teach you enough theory to know when you’re playing with fire, but we focus on applicability for the job to be done.

The journeys offer a personalized approach for professionals to grow their capabilities and to advance their careers. The blended learning journeys around product management consist of (competency) assessments, trainings, workshops, exercises, on-demand content, personal reflection, coaching, and consultancy.

The foundation of the product management journey is formed by the PPM framework. It’s connected to your product career framework and is implemented throughout the whole learning journey.

Want to get started? Take a moment to explore our website, learn more about our approach to Product Management, or schedule a chat about how we can help you, your team, or company to move forward.

Overview of all Competency Areas and Professional Product Management Training Modules