AB testing in the game with Python
Analyze a mobile game AB testing result and provide recommendations.
AB testing is a popular way to experiment with changes in websites, games, etc. We can see the result in a short time and make decisions accordingly.
I recently took several statistics thinking using Python courses and a course about designing and analyzing AB testing. As a result, I would like to apply my knowledge to the project.
Background& Problem Statement
The Dataset is about the mobile game Cookie Cats pops. At first, players download the game for free. As players progress through the game, they would encounter gates that ask them to wait some amount of time to enter the next game or make purchases to avoid waiting. As a result, the timing of placing gates is an important decision to make.
The Goal
Currently, the first gate was at level 30. The company wants to know the effect of the first gate at level 30 and level 40. Would the change affect player retention?
Process
1. Understand the dataset
2. EDA and data visualization
3. Hypothesis testing — bootstrapping
4. Result and business recommendation
Following, I will briefly take about key findings and takeaways of the project. You can access the full project code here.
Exploratory data analysis
The dataset contains 90189 players' data. There are two versions: gate_30, gate_40, total games the player has played, and if the player logs back the game after day1 and day7.
User count vs. game rounds
It is clear to see that most users play around 1–100 games. Interestingly, there are about 4000 people who just download the game and never play.
Gate 30 version and gate 40 version’s game rounds distribution looks similar, and the retention rate of two groups Gate 30 is 44.8% and 19%, gate 40 is 44.2% and 18.2% respectively. It seems similar, and are we confident about the difference? Let’s deep dive into this statistically.
AB testing
Perform AB testing to see if there is a difference between the two versions.
Hypothesis
H0: There is no difference between the gate 30 and gate 40 version.
H1: There is a difference between the two versions.
Use retention means a test statistic.
I would like to use bootstrapping to repeatedly resample the data 10000 times and get the 1-day and 7-day retention rate for the sample.
Before that, I would like to share some useful functions for creating bootstrap replicates I learn from the DataCamp course.
The first function is to random choice value from the list and re-sample it. The second one is using for loop to create replicates and add to bs_rep list. You can specify the number of random samples you want to replace func with your test statistic function.
Estimate the difference in the retention rate between the two groups and report a 95% confidence interval.
Below is the code snippet, and I would share the result at the end.
The result
Day1’s difference means is 0.6%, and the probability of difference is 96%.
Day7’s difference means is 0.8%, and the probability of difference is 99%.
Hypothesis test
The p-value of day1 is 0.0065, and day7 is 0.0069. As a result, I rejected the null hypothesis.
Result
Based on the result of the hypothesis test, there is a significant difference between the two groups. The retention rate for gate 30 is higher than gate 40 for both day one and day seven. The result tells that we can confidently believe that day one and day seven retention rate of version gate 30 is higher than the version of gate 40. In short, the company should not change the gate level to 40.
Takeaways and recommendations
- The company should not move to gate 40 since 7-day retention is higher when the gate is at level 30.
- The retention rate drops from 44% to 18% within six days. The company should figure a way to maintain their player based. It might be players get bored at day3 or day4 and decide to leave the game. To improve the retention rate for the first week, the company can provide some incentives such as free gifts are add interesting challenges, etc.
I hope you enjoy reading this!
