Computing incremental sales at Untie Nots
Part 4: Using a more robust approach. A general overview
Looking for the start ? Click here for part 1
For transparency and clarity towards our non-technical clients, we have only looked at solutions that are easy to understand and implement so far. In this section, we will look at the method that we rely on internally to determine the incremental sales reliably: causal inference with doubly robust estimators.
Causal inference is the process of determining independent effect of some intervention on the outcome. It is widely used in social sciences and medicine when randomised experiments are not feasible or unethical.
It is also very useful in marketing, where we often have access to only observational data and still need to measure the true effect of a marketing campaign. We also don’t need to spend time putting aside a control population, everyone can be exposed to the campaign.
In this blog post, we will only look at the high level ideas, if you’re interested in the mathematical details, check out the next blog post.
How it works
Like the name suggests, we will use two machine learning models in order to correctly compare the player and non-player spendings.
Rebalancing the dataset
We can’t simply compare the player and the non-players as the populations are not homogenous. People who come to the store more often are more likely to play, since they have more to gain; this means that players have a higher base sale than non-players and we cannot assume that their future spendings will be the same. The goal of the first model is to re-weight the customers so that the populations become comparable.
For example, if there are 20% high spenders among players, but only 10% among non-players, then we will count the high spending non players twice.
For each customer, we know whether or not they participated in the campaign; we also have access to all their purchases and characteristics before the campaign. With this information, we can train a machine learning model to predict the likelihood that a customer will play. However, this relies on an important assumption: there is overlap between the non-players and the players; meaning that for the customers with the same pre-campaign characteristics, some will play, some won’t. In practice, this is always true when looking at marketing use cases.
Once our model is trained, we can apply it to the entire population. Then we will predict the propensity score for each customer , eg: the probability that they would participate in the campaign. We then weight every player and non-player based on this score.
Estimating the alternate realities
When we evaluate the incremental sales, what we are really comparing is how much the customers would have spent had they played against how much they would have spent had they not.
By computing an incremental score, what we are really asking is how much the customers would have spent if they played in the game or not. Unfortunately, in the real world, we can only know one of the two values: either the customer player or he did not.
The goal of the second algorithm is to predict the second theoretical value. To achieve this, we train a machine learning model on all the exposed customer to predict their spending during the campaign. As input, we use their spending patterns before the campaign, their customer attributes AND the treatment value (the fact that they played or not).
We can then apply the model to each player and switch the Treatment value. For each customer, we can now compare, the real sales values and the estimated new one.
Combining the two models.
To compute the incremental sales, we combine the outputs of the two models. For every customer, we estimate their propensity score and their alternative spending value. For every player, we can compute the incremental spending brought by the campaign, and then weight those value with the propensity score in order to get the Average Treatment Effect on players. We simply need to multiply by the total number of players to get the incremental sales brought by the campaign.
So why is it called doubly robust ?
One of the key advantages of this strategy is that it gives an unbiased estimation of the incremental sales if at least one of the models is correct. Given the messiness of real world data, this makes our solution more able to isolate the participation in the campaign as the only factor in the incremental sales.
Just like in the twin method, we will use a validation period to make sure our models are correct, and adjust for any leftover bias.
Next part
This has been a high level description of the method, if you want more details on what happens exactly under the hood, check out part 5.
