The Importance of Stratification!

In my last post, I went through an experiment that showed us how variance in the data across time can cause issues in drawing inferences from an A/B test that has been run over a long duration.

Time, however, is not the only factor that can lead to misplaced inferences. More often than not, there are other variables that can lead to false inferences being made and that is when the concept of Stratification of the data comes into play. Let’s go through this example of an A/B test and see how Stratification can get us to the right result.

Looking at the Data

The data consists of 10000 data points consisting of the customer ID, Age classification of the people visiting the website, the time of the visit, and the conversion Status.

Let us go ahead and perform our analysis on the Data.

Data Analysis

Experiment 1

The first thing a company would like to check is the comparison of the click-through rate between Group A and Group B. So let us do that:

From the graph, we see the click-through rate for B is slightly lesser than that of A. Just on the basis of this the company could assume that B does not work as well as good as A on this population. However, would that be the right decision to make? Let’s do some more analysis.

Experiment 2

The dataset gives us one factor — Age. Let’s perform a basic analysis to see what it represents within the data set.

We see that the number of Old people in both Group A and Group B is much more than that of young people. This could be leading to the age-old case of oversampling for a particular population. However, could over sampling be leading to a misplaced inference?

To answer this question, we sample from our dataset to now have an equal number of old and young people in both Group A and Group B.

We then perform the same experiment as before to compare the click-through rates of both Groups.

What happened here? Suddenly, B is starting to perform much better than A. But what could have led to this not being seen across the whole population? Why does sampling over an equal number in both age groups lead to a flip in the initial assessment?

Let us dive deeper into each Age group and find out what is happening.

Experiment 3

Let us now split our initial data set into 2. One with only the young people and one with only the old people. After splitting them into two datasets, let us perform the 1st experiment on both of these datasets separately.

Old People

Young People

Aha! There we have it. The reason our inferences were all misplaced was that ‘A’ works better with the Old people whereas ‘B’ works much better with the Young people.

Based on this finding, we can draw our final inferences.

The reason the first experiment showed us that ‘A’ is better overall is that the number of old people is much higher than young people.

The reason the second experiment showed B to be better is that B works much better than ‘A’ for young people than A does than ‘B’ for old people.

This is why the Stratification of data on the variables present is such an important step to perform. It allows us to gain an insight into how a particular treatment has an effect on different layers of the population and we can avoid getting into the trap of a confounding variable, in this case, Age.

Thank you for reading through!

— Abhijit Krishna Menon

You can find the Jupyter Notebook where these experiments were performed here!

Originally published at https://www.abhijitkmenon.com on July 30, 2020.