The Logic Behind A/B Testing with Sample Python Code
In that topic, we will consider the evaluation of the advertising methods of a large company.
The data we have includes the new advertisement proposal method of a large company and the old advertisement proposal method. Thanks to this data, the company wants to compare the old method, that is, the current method, with the new method. Here, we will look for a conclusion about which method is more successful. In this way, the company can continue with the method we decided and get more profit or Impressions/clicks.
Now we want to understand the performance between methods.
Should we just “observe” the performance of these two methods and conclude?
Of course no, we should choose the more methodical/statistical approach to compare the performances between these two methods, so what is this approach?
As can be understood from our topic, the statistical method we will try is the A/B Test.
So, what is this A / B Test?
Today, the A / B Test is “a random online experiment that consists of two variants A and B”. This test quantitatively compares two samples with a single “measure of choice” in the assessment to determine whether there is any statistical significance between them.
However, we understand that the job is essentially a modern online adaptation of statistical experimental frameworks called Hypothesis tests.
So how should we apply the test?
0.Importing libraries and necessary data
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import statistics
from scipy import stats
from scipy.stats import shapiro
import statsmodels.stats.api as sms
import warnings
warnings.filterwarnings("ignore")control_data = pd.read_excel("Lectures/Week 5/Dosyalar/ab_testing_data.xlsx", sheet_name="Control Group")
test_data = pd.read_excel("Lectures/Week 5/Dosyalar/ab_testing_data.xlsx", sheet_name="Test Group")
control_data["Group"] = "A"
test_data["Group"] = "B"
is_any_outlier(control_data, "Purchase")
is_any_outlier(test_data, "Purchase")
control_data.head()
test_data.head()
1.Preferred Metrics
Normally we should look at which advertising method is expected to bring us a conversion, ie “Click Through Rate”, but we do not have any data directly presenting it, so we need to do Impression / Click. Before that, we will make an analysis of the Purchase column.
If we define the clickthrough rate (CTR) as phat, it will be phat = Impression/click.
We’ll set a Significance Level, a will be the metric we’ll take from p-values and use to compare hypotheses. I prefer the generally accepted value of 5%. = 0.05
2.Applying the Hypothesis
Before applying the hypothesis, we have to do an assumption check first, here are the steps we will consider;
- Assumptions of normality
- Homogeneity of variance
2.1 Let’s adjust our table for Assumptions of normality
AB_test = control_data.append(test_data)
test_statistics, pvalue = shapiro(AB_test.loc[AB_test["Group"] == "A", "Purchase"])
print('Test Statistics is %.4f, p-value = %.4f' % (test_statistics, pvalue))
test_statistics, pvalue = shapiro(AB_test.loc[AB_test["Group"] == "B", "Purchase"])
print('Test Statistics is %.4f, p-value = %.4f' % (test_statistics, pvalue))
Using these hypotheses, we can arrive at a decision, if the p-value is <0.05, then H0 is Rejected, if the p-value is> 0.05, then H0 cannot be rejected.
When we apply this test to our dataset, we cannot reject the H0 hypothesis because our p-value is not less than 0.05 in both A and B groups. Therefore, the normal distribution assumption is provided.
2.2 Assumptions of homogeneity of variance:
First, let’s make our hypotheses about the homogeneity of variance.
stats.levene(AB_test.loc[AB_test["Group"] == "A", "Purchase"],
AB_test.loc[AB_test["Group"] == "B", "Purchase"])
Here, we applied our test with the levene function to our test A and B groups over the Purchase variable. We got the value of p-value = 0.10 from our test, since this value is not less than 0.05, H0 could not be rejected, and variances are homogeneous.
Since we see that our assumptions are satisfied in this step, we will conduct two independent samples t-test (parametric test).
3. (AB Test) Independent samples t-test
Since we see that our assumptions are satisfied in this step, we will conduct two independent samples t-test (parametric test).
test_statistics, pvalue = stats.ttest_ind(AB_test.loc[AB_test["Group"] == "A", "Purchase"],
AB_test.loc[AB_test["Group"] == "B", "Purchase"],
equal_var=True)
We cannot reject our H0 hypothesis because our p-value is 0.34 (p value> 0.05) as a result of the t-test we have done.
Long story short, when we examine the values of the two methods we have, we can say that there is no need to process the new method since there is no statistically significant difference between the two group averages. However, since the data set we have is not large enough, it will not be right to make this decision so soon. Perhaps the number of observations can be increased and the result can be more accurate.
# Bonus Now let’s take a quick look at Click-through Rate.
Click-Through Rate (CTR)
It is the ratio of users who visit the website, see the ad and click the ad.
Clicks / Impression
control_data['ctr'] = control_data["Click"]/control_data["Impression"]
test_data['ctr'] = test_data["Click"]/test_data["Impression"]ogi_AB(control_data,test_data, "ctr")
Group A has a higher value compared to the overall rates seen for now. At first glance, we see that the click-through rate is in favor of the control group. So while the ad is showing, the rate of site visitors clicking seems to be better in the current system.
But this will be a simple view, will there be a statistically significant difference here, so we need to do our AB test.
I would like to express my sincere thanks to Vahit Keskin and my mentor Atilla Yardimci who helped and taught the completion of this project
The notebook associated with this article is on Github if you want to follow along.
