Metrics, Metrics Everywhere: Elevate Product Development with High-Quality Success Measures
Early in my career as a new analytics manager, I had the opportunity to sit down with a data leader at one of the world’s largest tech companies (hint: it’s a social network). As its analytics leader, he was responsible for overseeing hundreds of analysts, data scientists, and data engineers. Our conversation ranged from team building, collaborating with key stakeholders, and eventually turned to metrics, especially the all-too-common ask to build a dashboard for this and a dashboard for that. His guidance was simple, and has continued to stick with me:
Use metrics to drive actionable insights, not just dashboards!
While seemingly straightforward, this advice has served as a powerful reminder that data is only as valuable as the actions you take. I’ve seen this first-hand during my years at Uber as well as my current role at Klaviyo. Capturing and storing data is easier than ever today. Given this proliferation of information, the initial key step to driving impact is prioritizing *what* to measure, not whether or how to measure it.
In a series of posts, I will share the ways that I’ve been able to bring data to life and drive outcomes. Today’s entry will focus on how to select and organize metrics to most effectively derive actionable insights. Future posts will explore goal setting, experimentation on feature launches, and data storytelling.
Since I’ll be referencing actual examples, let’s start with a quick primer on Product Analytics at Klaviyo. Our Product organization is structured as a number of cross-functional Product Areas, each with distinct, long-lived missions. Product analysts collaborate closely with product managers, engineers, and designers and help drive the quantitative needs of the Area (understanding key trends, experiment analysis, modeling, deep dive explorations, etc). The use of metrics is one way to build a foundational understanding of past and current performance as well as looking to the future via goals and targets.
As mentioned above, one of the first challenges is effectively narrowing down an often very broad set of possible measures.
In order to help guide metrics prioritization, I’ve found it useful to breakdown metrics into two primary types, each of which we will explore in more detail:
- Business impact metrics
- Customer focused metrics
BUSINESS IMPACT METRICS focus on the overall outcomes that are critical to a company’s success. They often are expressed in currency and may appear on the P&L. Common examples include revenue, profits, and cost savings.
For newer products, the focus is often on gaining traction in the form of revenue. One example is the Mobile team at Klaviyo. We launched an SMS product at the end of 2019, so SMS Revenue is a valuable metric to understand our progress to date. Another example from my Uber experience was our introduction of new car types (UberX, SUV, UberPOOL, etc). Tracking the % of Revenue for each car type within a city provided an excellent viewpoint into the total share of wallet from riders.
For more mature products, the focus may shift towards efficiency and profitability. At Klaviyo, several teams track ratios related to Owned Revenue, a measure of the revenue our customers earn from their owned marketing channels (email, web, and mobile). Relevant metrics include Owned Revenue per message, Owned Revenue per campaign or flow, and % of overall Owned Revenue. More to come on Owned Revenue later in this post. Another example comes from my experience on the Rider Growth team at Uber. For all accounts that signed up, we closely tracked Blended Cost per Acquisition across our channels (referrals, paid, organic, etc). This measure was especially useful in comparison with Lifetime Value to better understand cohort performance.
Note that not every team will directly influence one of these primary business metrics; in that case, it is fine to focus on customer metrics that indirectly ladder to a business outcome. For example, a team may focus on increasing a feature’s Monthly Active Users (an engagement metric), which indirectly drives revenue and profits (both business impact metrics). More to come in our next section.
CUSTOMER FOCUSED METRICS track the journey of an end user as he or she interacts with the Area’s products and/or features. For many teams, this means the actual customer of the company; however, it can also refer to internal customers for back-end or platform teams. Either way, a Product Area should work to jointly optimize both growth in usage of its product or features and a delightful experience for customers. We will explore four themes within customer focused metrics, two that can be classified as growth and two as delight.
We begin at the start of the user journey. The first step is customer adoption, which tells us whether users are discovering your product or feature. One common way to measure adoption is through a new user funnel, tracking the conversion rate for each step from initial intent to success. Additional metrics for adoption and the following themes can be found at the end of this section.
Post-adoption, we track engagement to better understand continued usage by existing customers. A popular measure of engagement is Active Users, which can be tracked monthly (MAUs), weekly (WAUs), or daily (DAUs) depending on the use case frequency for your product or feature. Beyond actives, it is also important to monitor retention of customers after using your product, often at 7-day or 28-day time windows.
Together, adoption and engagement combine to capture overall growth in feature usage.
While adoption and engagement metrics are useful, they don’t tell the entire story. It’s possible that customers are using your feature while actively disliking it. This is why it’s important to also capture measures of delight. We start with the experience, where we assess customer satisfaction. Two ways to measure the user experience include tracking sentiment (NPS, CSAT, etc) and customers proactively surfacing issues (in the form of support tickets).
Finally, reliability is our fourth theme which looks at stability of use. Reliable products or features are ones with infrequent bugs and outages. Measures of reliability are especially important for platform teams focused on uptime and meeting SLAs.
Bringing everything together, you can see how visibility into the four themes of adoption, engagement, experience, and reliability can provide a more comprehensive view of the customer’s interaction with your product or feature. At Klaviyo, we strive to track at least one metric from each of these four groupings. Product Areas with significant end user touch points may emphasize growth, experience, and delight, while more back-end focused teams often prioritize reliability metrics.
To close this section, I’ve included a set of measures worth considering as you assess Product Areas and/or new features across our four themes.
While a framework for selecting the right metrics is useful, this alone is not enough to bring your data to life. In this second half of the post, we will go deeper on several features that will enhance your ability to translate metrics into action.
North Star Metrics
Let’s start with north star metrics. Each Product Area denotes 1–2 of their set of metrics as north stars to even further prioritize what is most important.
“A north star metric is the key measure of success for the product team in a company. It defines the relationship between the customer problems that the product team is trying to solve and the revenue that the business aims to generate by doing so.” (from Amplitude)
North star metrics are extremely useful for the following reasons:
- Providing organizational clarity and alignment
- Communicating impact and progress to the rest of the company
- Holding team accountable to an outcome
Examples of north star metrics include:
North Star: Daily Active Users
Facebook emphasized this engagement metric, striving to get its users interacting with the app every day. By focusing on DAUs, the Facebook Growth Team identified that new signups who reached 7 friends in 10 days were far more likely to remain active. This insight greatly shaped their strategy for onboarding new users (read more).
North Star: Messages Sent (specifically 2,000 within an organization)
Slack found that its customers fully realized the value of its product upon reaching 2K messages; retention soared above the threshold (read more).
North Star: Weekly Trips
With its two-sided marketplace, a completed trip represents a successful match between the supply- and demand-side. An emphasis on trips led to a joint focus on improving the driver and rider experiences to ensure proportional growth. Note that Uber has since transitioned to Monthly Active Platform Customers, partially to better capture its other business units (such as Uber Eats).
North Star: Owned Revenue
Our mission is to help our customers grow, as our tools allow for automated, dynamic segmentation and improved personalization of messages to end users. As our north star, we track the revenue our customers generate via owned channels (email, web, and mobile).
Product Areas have a narrower scope than an entire company, so its north star metrics may be more specific than the examples above. You may want to consider laddering up your Area’s north star metric to the company’s primary metric. Note that your north star metric can change as priorities shift, but ideally it is consistent for several quarters (if not years). Lastly, rallying a team around a north star can be a unique motivator, as well as an opportunity to storytell on progress and trajectory.
Goal Setting
Beyond north stars, we strive to set goals for as many metrics as possible, as tracking performance against a forecast can inspire greater impact and better capture unexpected variance worth exploring. This topic deserves its own blog post; stay tuned!
Segmentation
Focusing your metrics via segmentation is a powerful way to isolate the performance of customers with specific characteristics. Common segments include customer size, channel, geography, industry, and device; each business will have its own critical segments. Depending on the near-term roadmap, you may want to assess one or more metrics on specific segment(s).
Healthcards
Lastly, visualization matters. At Klaviyo, each Product Area tracks their most critical metrics in a one-page healthcard. We target 8–10 metrics and refresh the data regularly to understand the trends. Reviewing these healthcards are an important part of our weekly product team meeting. Klaviyo healthcards aren’t just charts and numbers; there is a section at the top where we capture 2–3 highlights or opportunities to improve. This is an important final step, as it forces you to truly consume the data and identify what is really happening.
See below for an example healthcard which brings together all the concepts about metrics that we have discussed:
To conclude, metrics can be a powerful way to track progress and inform the future product roadmap. Don’t settle for “set it and forget it” dashboards; push yourself to understand the role each metric plays in capturing the broader vision. Prioritize via north star metrics, set ambitious goals, and consider your visualization to ensure maximum visibility across key stakeholders. With these principles in mind, you will be well on your way to driving actionable insights!
