Choosing the Right Metric

When applying data science to business problems, it is important to understand the problem space, start simple, and most importantly choose the right metrics to optimize against. KPIs are powerful tools for aligning stakeholders around a goal and measuring progress. However, if you choose the wrong metric you may create unnecessary friction in your organization and fall short of your goals.

A Simple Example

Let’s imagine you are a data scientist tasked with helping to create a system that will target an email campaign to customers with an offer for a specific product based on some information gathered as the customer passes through your sales funnel. You need to determine what metric to use for this optimization task. Your first inclination may be to build a classifier that predicts which product each customer would select (if any) based on the historical sales data; let’s call the class probabilities from this model the offer take rate.

Now that you have a prediction for the offer take rate, what can we do with it? First, let’s assume we don’t care about unit economics we just want to to maximize the conversation rate of our funnel such that we acquire as many customers as possible. In that case, we can simply present each customer with an offer for the product they would most likely purchase based on historical data. This will send the offer to each customer for the product they are most likely to purchase and should lead to the highest conversion rate. In some situations this simple approach may be sufficient, however, often other factors at play may make acquiring as many customers as possible undesirable. How do you know, for example, that these transactions are profitable?

What if instead of maximizing the conversation rate, we wanted to maximize revenue? Let’s assume that we have three products a “Base” model for $30, a “Pro” model for $50, and an “Ultra” model for $75. For a particular customer, the predicted offer take rates are No Offer 60%, Base 20%, Pro 15%, and Ultra 5%.

Now, we have some additional information that we can use to calculate the expected value of each offer presented. Base 20% x $30 = $6, Pro 15% x $50 = $7.50, and Ultra 5% x $75 = $3.75. From these numbers, we can see that if we want to maximize revenue presenting the offer that maximizes the take rate does not have the highest expected value. If we have to choose a single option, choosing to present the Pro version will increase our expected revenue by 25% over the Base option (which has the highest take rate). We don’t have to stop at revenue, if we can model the COGS for each offer we can maximize profit for each transaction instead. Now you can see how tweaking the metric you are optimizing ensures that you are generating profit, which can be balanced against the cost of acquisition.

Customer Lifetime Value

Let’s look at another strategy, rather than focusing on maximizing the profit of each transaction, let’s consider a business that is interested in building a long-term relationship with each customer. In this case, a model that maximizes the value of each transaction may drive down customer retention and satisfaction rates. Having a longer-term view of your relationship with a customer also helps to understand the full return on investment for customer acquisition cost. If you consider all future interactions with each customer you can determine the future costs and revenues generated from those interactions to measure the value created over the lifetime of that customer, the Customer Lifetime Value (CLV).

Now let’s consider a situation where you have a freemium offering designed to give customers certain features for free to demonstrate the capabilities of your platform and provide an upsell opportunity to a paid option. If you look at this from a purely transactional perspective you would not want to make an offer for a freemium service to any customer because that transaction would have a negative expected value. Let’s instead think about these transactions as the start of a longer-term relationship with the customer.

For a freemium service, you can then model the likelihood the customer will convert to a paid plan as well as the duration they will pay for their subscription and the pricing strategy for that service over time. This allows you to calculate the CLV for that customer and you can determine which customers will be the most valuable and where we should invest our acquisition dollars to attract those customers. This metric will provide a means to gauge the value of the freemium offering and to target customers who are likely to convert after some time on the freemium plan.

A carefully constructed CLV model is a powerful tool for your business because it is a metric that can account for all the activities and interactions you have with your customers as well as the COGS associated with that customer. CLV provides a useful metric for:

• Acquisition: CLV can be used to measure the performance of your acquisition funnel across channels and regions. By optimizing for value creation rather than customer acquisition, you can ensure that you are driving profitable growth. CLV ensures that you are acquiring customers that are easier to retain.
• Retention: Are you retaining the right customers? CLV can be used to ensure that your retention efforts are focused on the most valuable customers and that customers with negative CLV are not retained.
• Pricing and targeting: Targeting customers for the product that will maximize CLV, and pricing them according to their individual COGS can be a huge competitive advantage.
• Cross-sell/up-sell: CLV models can also account for cross/upsell opportunities as well as relationships with customers that involve many transactions over time.

Conclusion

Data teams need to understand how a business functions to ensure that you have the right data and the right metrics to define and execute your strategy. Businesses must consider how the metrics set for each team will complement each other to align those teams and ensure that the business as a whole is healthy and profitable. In this example, CLV is used as a metric to align the goals of acquisition and retention to maximize the profitability of each customer. Every business is different and strategies will change based on internal and external factors, it is important to understand how your metrics should change as your strategy evolves.