Jul 16, 2019
Selecting the Right User Metric
Previously, we discussed how to define success by setting the right goals and metrics. Goals help you define and monitor success, and they connect your mission to your strategy, roadmaps, initiatives and tactics by tying the single metric you care about most to a clear target and time frame. Setting the right metric is crucial for a product to succeed but picking the one metric that matters most can be challenging. In this post, we’ll address how to select the best metric to drive your product’s success.
What is the one metric that matters most to the success of your product? The answer is different for every company. For Facebook, it is active users; for WhatsApp, it is the number of sends; for eBay, it is gross merchandise; for PayPal, it is total payment volume. Once a top-line metric has been identified, it can be used to evaluate and manage the health of your product by setting criteria for success around it, monitoring it, understanding what drives it and obsessively pushing it — and your product — in the right direction.
Each class of problems has unique challenges in identifying the best metric. For the purposes of our discussion here, let’s consider products where the number of active users is the most important metric. Within that category, is it better to track intraday users, daily active users (DAU), weekly active users (WAU) or monthly active users (MAU)? Generally, all of these metrics are correlated, and picking any one of them would be reasonable. But is one of them better than the others? There are three considerations for picking the right metric:
- Vision for the product: How do you envision the product being used, and how will that change over time? Do you anticipate weekly product usage? If so, WAU would be the best metric to drive success. But if you also anticipate that people will transition from desktop usage to mobile usage, you might want to consider selecting DAU as your goal metric since mobile platforms tend to increase usage frequency.
- Current product usage: You might have a vision for your product, but how are people actually using your product right now? If people are using your product multiple times a day, then it makes sense to select DAU or an intraday metric (number of sessions or hourly active users) your most important metric.
- The competition: How are competitors’ products being used? Do people use competitors’ products on a daily basis but your product is used on a weekly basis? If so, use DAU as the metric because your product will not succeed in the competitive environment as people are engaging it with less. Setting a daily usage goal may force product decisions that you otherwise may not have done.
How a product gets used is crucial for setting the right metrics, so let’s explore that further.
Product Usage
The usage of any product depends on the product’s intended purpose. A product designed for use in the workplace is unlikely to have many users over the weekend, whereas a social product is likely to be used uniformly every day of the week. Ultimately, the usage of any product comes from a mix of people who are using it in different ways across different devices and platforms. Understanding this mix is essential to defining the right goal metric for your product. Here, we provide an outline of the different types of product usage — intraday, daily, weekly, monthly and quarterly.
Metrics for Intraday Usage
Products that people use multiple times a day have intraday usage. As product usage increasingly transitions from desktops to mobile phones, the number of products with intraday usage has grown. Some of the leading social, search and browser products generally fall into this category. If your product is designed for intraday use, there are two key metrics you can use to drive success:
- Hourly active users: Define an hourly active users (HAU) metric and check how many times a user registered as an HAU within a day. The greater the number of HAUs, the more your product is being used intraday.
- Daily sessions per user: If the number of daily sessions per user within a day is high, then the product is likely to have high intraday usage.
The number of product sessions per day is a good indicator of how frequently a product gets used. Messaging products such as iMessage, WhatsApp, Facebook Messenger and KakaoTalk are all heavily used multiple times a day. Social products such as Facebook, Instagram and Snapchat are also used multiple times a day, although not as frequently as messaging products. There are also products that get used multiple times on some days but not every day. Travel, dating and navigation products all fall under this category of sporadic daily use. Figure 1 shows the number of sessions per user, per day for different products. As you might expect, the dominant messaging, social and browser products have high levels of daily sessions. In contrast, most people do not shop or check the weather multiple times a day, every day. All the figures and tables in this document were derived from App Annie.
Metrics for Daily Usage
As the name suggests, a daily product is used on a daily basis. Many mobile products can reach high daily usage as well as relatively high hourly usage. there are three key metrics to track:
- L5+/7 This tells you what percent of users are active more than four days of the week. If a large percent are L5+/L7, the product should be tracked by daily usage. Note that L1+/7 is the same as WAU.
- L21+/28 This tells you what percent of users are active more than 20 days of the month. If a large percent are L21+/L28, the product should be tracked by daily usage.
- DAU/WAU. If DAU/WAU is 60% or higher, this means that people use the product more than four days a week, and the product qualifies as a daily usage product.
Let’s look at the DAU/WAU for some familiar products that have significant monthly usage. (Figure 2)
Samsung Experience Home has the highest DAU/WAU ratio. Samsung Experience Home is the official launcher that provides a user interface optimized for Galaxy devices, and it comes pre-installed on every Samsung phone. Unsurprisingly, the product’s DAU/MAU is so high because all of its users open their phones nearly every day. Many social products are all heavily used and are mostly daily usage products. Microsoft Outlook is an email client that is primarily used for work, so one would expect people to use it five days a week. The normalized DAU/WAU for Outlook is close to 84%, confirming that people are using email every day of the work week. All of these products are also products that are used intraday. To our knowledge, no company sets an intraday goal explicitly. Companies like Facebook have Time Spent goals which is another proxy for intra day activity. This is in addition to the daily active user goal that they have in place. There are complexities in measurement and operations that currently make intraday metrics less tractable.
As the chart shows, music apps (e.g., Pandora, Spotify), e-commerce products (e.g., Walmart, Amazon) and video products (e.g., Netflix, YouTube) are not used on a daily basis by the community at large. However, there are pockets of product users that engage with these products every day who can be identified with additional data such as L7 (the distribution of user usage within a week) or L28 (distribution of user usage within 28 days). Additionally, the chart shows that Google’s Calendar, Maps and Photos products are utility products that do not get used every day.
The variance in product usage among different users and their associated distribution means that it is very rare for products to reach a DAU/WAU of 90% and above. Therefore, a DAU/WAU of 60% suggests that there are people who use the product every day and others that use the product much less. It is this blend of the different groups that results in the aggregate DAU/WAU being around 60%. This rule of 60% as a threshold for identifying the best active user metric for a product is generally helpful across all use cases. In the case of HAU, we use a lower threshold of 50% based on empirical usage activity of some of the top hourly used products.
Metrics for Weekly Usage
For products with weekly usage, the key metric to track is WAU/MAU. If the average user opens the product between two and three weeks of the month, i.e. WAU/MAU is at or above 60%, then the product could be classified as one with weekly usage.
For the same products discussed in the previous sections, we have calculated the WAU/MAU (Figure 3). WAU/MAU can be thought of as the percentage of weeks in a month during which people use the product repeatedly. Again unsurprisingly, the chart shows that social products have a very high incidence of people returning each week. Microsoft Outlook is used every week as well. However, since these products have a high DAU/WAU, it makes sense to identify them as daily products, and to set daily active users as the key metric.
In contrast, Spotify and YouTube are not used daily but are used nearly every week. As a result, these would qualify as weekly products rather than daily products.
Netflix, Walmart and Amazon are not used daily, but they are used roughly 2–3 weeks in a month. Even though they do not always meet the rule of 60%, from an aspirational perspective it makes sense to set a WAU goal for these products rather than a MAU goal.
Metrics for Monthly and Quarterly Usage
For products with monthly usage, the key metric to track is MAU/QAU. If the average user open the product two of three months, so that MAU/QAU is at or above 60%, then the product is a monthly usage product. Similarly, QAU/YAU of 60% would be a quarterly used product (YAU is the number of yearly active users).
Many products are used on a monthly basis. As shown above, Venmo and PayPal are regularly used but not frequent enough to qualify as weekly products for a majority of users. The MAU/QAU for Venmo is 67%, and for PayPal it is 78%. This means that these two products are actively used by the community on a monthly basis. Similarly, one can identify companies that are use quarterly and not monthly.
Developing a Usage Metric Framework
Understanding hourly, daily, weekly, monthly, quarterly and yearly active users helps us develop a framework for classifying product users. For every product, one can identify HAU, DAU, WAU, MAU and QAU, and this should be illustrative for understanding the usage patterns of a product’s users (Table 1). Generally speaking, if a product is predominantly used daily, it is also likely to be used on a weekly and monthly basis as well. As a result, it is sufficient to find the first value that clears the 60% threshold (or 50% in the case of HAU) as we move from left to right in the framework.
Next, we can apply the framework to actual companies. In the table below, we constructed the DAU/MAU, WAU/MAU, MAU/QAU and QAU/YAU ratios among users in the United States for ten technology companies (Table 2). Facebook is used daily, with a DAU/WAU ratio of 82%. Amazon is used around three weeks per month, making it a weekly product. Products like Walmart, Uber, Netflix, Venmo and PayPal are used on a monthly basis, but not on a daily or weekly basis by most people. Groupon and eBay are used a little less than two quarters per year, and should aspire to become at least quarterly products.
Additional Metric Considerations
Some additional factors to consider in metric selection include:
- Distributions, mix effect and aggregates: In aggregate, a product’s usage is derived from a mix of different types of users. Almost all products have people that use the product frequently and others that do not. The user ratios we’re discussing here provide an aggregated percentage of how frequently people use a product. However, the details are hidden in the aggregates and a deeper understanding of the distribution will uncover additional insights. For example, Uber has a DAU/WAU of about 25%. If 88% used the product one day a week and 12% used it seven days a week, the blended average would be 25%. This means that the distribution of the user mix is important to understanding if the product is truly used on a daily basis by a significant segment of users. If the product is bimodal, pick the aspirational goal. In the case of Uber, choose the daily goal instead of the weekly.
- Ratio bounding values: It’s worth noting that DAU/WAU can never be lower than 1/7, DAU/MAU can never be less than 1/28 (where MAU is defined over 28 days), and WAU/MAU can never be lower than 1/4. As a result, the DAU/WAU of eBay (14%) is close to the lowest possible value, and indicates that nearly every eBay user engages with eBay once a week at the most. So, eBay is clearly not a daily usage product.
- Products change and usage ratios are slow: Usage ratios are slow-moving metrics that provide an overall snapshot of a product at any given time. As such, they do not always capture changes in usage. For example, DAU/MAU may not immediately reflect a shift in how new users are engaging with a product. A cohort retention metric would capture changes in new user behavior much earlier than DAU/MAU. If newer cohorts have better retention and older cohorts are resurrecting faster than churning, then DAU/MAU is certain to improve. It may make sense to change your product’s key metric if there are early indications of usage changes.
- Application to other types of metrics: The framework developed in this document is for active users only. Other types of phenomena, such as time spent, revenue, conversion rates, etc., require their own frameworks.
Takeaways
- Selecting the right user metric depends on the vision, usage and competition.
- Careful exploration of the usage will help determine the selection of the right user metric for success.
This work is a product of Sequoia Capital’s Data Science team. See the full data science series here. Please email data-science@sequoiacap.com with questions, comments and other feedback.
