To incentivize or not to incentivize — is it worth the cost?

Decision frameworks to analyze the effectiveness of incentives

At Digit we love to ship new features that help our members achieve financial health. While we always believe that these features are beneficial, it’s a natural part of the member lifecycle for a change to cause a subset of members to leave the platform (churn). One way we mitigate churn is with incentives, such as offering members a free month on the platform or another kind of promotion. These incentives can be helpful tools that can keep members on the platform long enough to recognize that a product improvement is valuable.

Incentives come with a cost. We want to know overall if the cost of the incentive will yield a positive result for Digit. After all, we want to remain financially healthy as a company, too. We can explore this from a data science perspective in two different ways:

• Long term revenue goals: Is the revenue gained from churn reduction greater than the cost of the incentive?
• Short term customer acquisition goals: Is the cost of preventing a member from churning less than the cost to acquire a new member?

In order to answer these questions, we perform an A/B test to measure the churn levels that will feed into our econometric models.

This post is about setting up a decision framework before initiating an experiment to measure churn. We can use it to determine the acceptable combinations of churn that justify the cost of the incentive. The decision framework also helps us set the sample sizes for the experiment.

We’ll share some formulas and code to help along the way.

A/B test setup

The initial A/B experiment has two purposes. The first is to discover the level of churn that will occur with the new feature rollout. The second is to measure the percent reduction in that churn when an incentive is offered with the feature rollout. If we are testing multiple incentives, they can each have their own variant in the test:

• Variant A: new feature rollout, no incentive
• Variant B: new feature rollout, with incentive X
• Variant C: new feature rollout, with incentive Y
• etc.

The percent reduction in the churn (R) is a negative relative lift between the churn from variant A (Cᴀ) and churn from variant B (Cʙ).

For example: If variant A has a churn rate of 35%, and the incentive offered to variant B reduces that churn rate to 30%, R=14.3%.

The sample sizes for the experiment will be determined after analyzing our chosen decision framework.

Decision frameworks

Before we begin the experiment, we want to know which combinations of churn from variants A and B (and the calculated reduction in churn, R, between them) will result in a positive return on our investment of the incentive. We do this by listing out all possible scenarios and what our corresponding payout would be.

The next two sections explore churn combinations that would justify an incentive for both long-term and short-term business goals.

Long-term goal: do we ever make a positive return on the investment of the incentive?

Digit’s business is subscription based, and we can project out the revenue expected from a cohort over time, including churn that would naturally occur. To understand the effect of different combinations of churn, we can look at the projection of our revenue as a function of time in the case where a new feature rollout caused churn, and in the case in which an incentive prevented some members from churning, but came with an up-front cost.

In the case of no incentive, revenue after M months is calculated as:

where N is the number of members in the cohort before the feature rollout, S is the monthly cost of the subscription, Cᴀ is the churn from variant A, and Cɴ is the natural churn level of the cohort each month without the introduction of any feature changes.

For the case where there is an incentive, we may have saved some members (from our definition of R in the section above, the percent churn becomes Cʙ=Cᴀ*(1-R)), but we have to subtract off the cost of the incentive, I,

The revenue gained from the incentive after M months would be

Using python we can code up a visualization of the revenue gained from the incentive after a certain number of months for different combinations of Cᴀ and R. The toy example (code and resulting figure) below looks at the revenue saved after 12 months assuming that there are 50,000 initial customers in the cohort, the subscription cost is $3 per month, there is a natural churn level in the cohort of 5% each month, and the incentive costs the company a total of $100,000.

In this toy example case, if group A from the A/B test shows a churn of 5%, it doesn’t matter what R is — 12 months after offering the incentive, the company will have lost money from the cost of the incentive when few members were going to churn anyway. If churn is very high but R is low, like the case of Cᴀ = 40% and R= 10%, the incentive didn’t prevent very many members from churning and revenue is lost. The long-term outlook for the incentive is only profitable if both Cᴀ and R are high, like in the case of Cᴀ = 30% and R= 40%.

Here, we would choose minimum sample sizes for an A/B test that would provide sensitivity to combinations of Cᴀ and R that provide a positive long-term revenue projection.

**Note — there are cases that could complicate the formula. For instance, if an incentive is offered only to non-churned members, the overall cost of the incentive will be dependent on the churn level of variant B. We won’t discuss those cases here.

Short-term goal: do we see comparable cost per acquisition to a new member?

If a company is actively marketing to gain new members, it is likely that there is an accepted cost per acquisition, or CPA (how much money on average a company is willing to spend to acquire a new member). If churn is expected due to a change in the product, the cost of an incentive which prevents some members from churning can be used to calculate a unit cost per saved member, analogous to the CPA. We can find out what the cost per saved member is as a function of Cᴀ and R from our A/B test and compare to our accepted CPA.

If the cohort that will see the new feature has N members, and we have measured churn levels Cᴀ and Cʙ from our A/B test described above, the number of members saved from churning, Ns, can be calculated as follows:

using our previous formula for the reduction in churn, R. This would give us a cost per acquisition of:

where I is the cost of the incentive.

Again, before even beginning an A/B test, we can develop a framework for understanding the result, which will assist in choosing the minimum sample sizes needed to see an effect that would justify the cost of the incentive.

The python code below creates a figure that shows cost per saved member (CPA, if being treated as a cost per acquisition) as a function of Cᴀ, given different levels of R. A horizontal line can be drawn across the accepted cost per acquisition to find the combinations of Cᴀ and R that would result in the incentive being justified.

** Note — the value of an existing user (measured by future lifetime value, perhaps) who is saved from churning may be greater than a new user who has never tried the product! The cost per saved user that a company is willing to accept in offering an incentive may be greater than the cost per newly acquired user based on the projected value of the two different cohorts.

Making decisions and the order of operations

As data scientists, it is our job to support decision making at the company through data, modeling, and experimentation. Through these decision making frameworks, we can help the company decide what goals we are trying to achieve by offering incentives and then set up the least invasive experiment possible to measure whether or not we will achieve those goals.

The smoothness of decision making around offering incentives is dependent on our order of operations. We want to work in this order:

• determine a list of potential incentives and estimate their cost,
• determine whether the goal is long-term revenue gain or short-term member “acquisition”,
• calculate the combinations of churn from an A/B test that would achieve the company goals,
• determine the minimum experiment sample size that would provide sensitivity to combinations of churn that would justify offering the incentive, and
• set up and run the experiment.

If we work in this order, after the experiment finishes running and the results are in, we already know what to do. If we detect a significant reduction in churn from the variant that is offered the incentive, we can compare to our long-term or short-term goal charts to determine if it is worth the cost.

Acknowledgments 💙

Thank you to Ravi Soni (Data Scientist at Digit) for doing this work with me!