Computing incremental sales at Untie Nots.
Part 3: Handling bias. Intuitive incremental methods.
Looking for the start ? Click here for part 1
Introduction
In the previous blog post we have discussed common industry methods to measure incrementality during marketing campaigns. From here on from this part, we will focus on the methods we use at Untie Nots to prove incremental effect. In this blog we will discuss simple and easily explainable methods: Direct incremental measurement and Twin method.
Lift ratio measurement
Remember that the lift formula takes the control group as our baseline and then compares it with the exposed customers’ purchases. It works under the assumption that control group was randomly selected, which guarantees that change in behaviour between two groups can only be explained by marketing campaign and not by other factors.
As we have seen before, it is hard to come up with good A/B testing protocol in our case. Thus, we decided to compare the players directly with the control group and accept that the baseline sales might be different among these groups.
We adjust the lift formula to take this discrepancy into account to get what we call a Lift ratio (LR):
Here, we are comparing the rates of change in behaviour during the marketing campaign and some fixed pre-campaign period between these two groups.
Notice that this formula does not necessarily requires control group to be randomized. Player group might be spending more on average even before the marketing campaign, but if the jump in sales during it will be higher than the jump in control group that should convince us that this can be explained mostly by the effect of the campaign.
This method is easily interpretable and can be used as a baseline for comparison with other methods. However, we can only hope to reduce the bias, but we probably won’t completely eliminate it.
As an example, imagine that people who tend to participate in the marketing campaign also spend more during this particular month due to some external reasons.
As a result, we would observe an incremental effect even if the campaign had never happened. In other words, if there exists a trend specifically for players, or specifically for control group, the formula will capture that as an incremental effect of the campaign.
In practice, however, we can assume the bias to be homogeneous among both groups so this effect is mostly negligible.
We use a few months of data for the baseline sales in order to capture long term purchase patterns. We have observed that the sales of players increase relative to the baseline before the campaign. This makes sense since people who are coming more often to a store than before are more inclined to participate in the campaign.
To account for this, we use a validation period between the baseline and the campaign. We then use the same technique to compute the lift for the validation period, and subtract it from the campaign lift. This is the worst case scenario for estimating the incremental, but it is an effective way of eliminating pre-campaign behavioural changes.
Thus after debiasing the formula will look like the following:
Twin method
We have discussed previously how non-representative control group prevents us from claiming that the increase in sales comes solely from the effect of our marketing campaign.
What if we could find the subset of customers in the control group, whose behaviour would be the closest to those in player group? This is exactly the idea behind the twin method.
We first define features that will define similarity, like weekly spends, frequency of visits, contactability, amount of received discount, etc. Then we will match each customer from the player population with the closest (in terms of behaviour) customer from the control group.
So what do we mean by closest? It is not clear when we talk about boolean characteristics like contactability and continuous ones like sales amount.
We took inspiration from the RFM strategy and divide our population into groups based on their customer characteristics (they depend on the retailer, but are usually opt-in, status, etc.) and their quantiles in frequency and discount over sales percentiles.
In each sub group, we then construct a vector space with the weekly spendings during the pre-campaign. For every customer, we match him with the closest control customer, in terms of euclidian distance, from the same sub group. Note that two players can share the same twin.
For each customer, we will have a twin from the control group, who matches his behaviour and characteristics as closely as possible before the start of the campaign. Thus we can think of the players and this new twin population as homogenous groups, and any difference we will obtain afterwards will be solely explained by the effect of the marketing campaign.
To be sure that we are not cherry-picking the control group so that they match just for the pre-campaign, we use the same approach as with the lift ratio. We introduce a validation period between the pre-campaign and campaign. This period is used to control the bias between the two groups as any difference will be removed from the final incremental.
In practice, it produces roughly the same results as the lift ratio, but has a few added benefits. This strategy is still intuitive to understand and has a nice visual representation of the effect.
It has, however, several problems:
- First of all, similar to our previous method, there is often a residual bias that we can’t correct. The use of the validation period gives us a worst case scenario.
- Second of all, we now have to obtain a big and diversified enough control group so we can find the closest matching customer for each player. Again, this would mean that we need to prevent more people from participating.
Next part
In the next part of the blog we will explain how we adopted the methods from causal inference field to compute incremental sales.
