Mock Research Proposal + A/B Test Analysis

Photo by Dana Tentis from Pexels

Note: This post, an assignment for a data science course, consists of a mock research proposal for a real-world A/B experiment followed by an analysis of the resulting data set. Product and company names have been changed. The data is real and has been sourced from kaggle.

Timer Gate Research Proposal

Background

We here at K’Netic Games value continuous improvement and innovation, whether with regards to our work flows, our customer relationships, or our product quality. As such, when reviewing our product lineup, we make every attempt to identify improvement opportunities and act appropriately by proposing actions we can take to realize those improvements.

Problem Statement

Brownie Bears, one of K’Netic Games’s most beloved games, remains popular in both the App Store and Google Play; however, our researchers have identified an opportunity to increase user retention with a simple change.

Figure 1: Brownie Bears Retention Rate Distribution, Previous 6 Months

We can see in the distribution plots in Figure 1 that the Day 1 Retention Rate centers around 44.7% while Day 7 Retention Rate has a mean of 18.99%

Figure 2: Brownie Bears Retention Rate by Day, Previous 6 Months

Figure 2 shows the retention rate per day for the last six months. The rates have not deviated much from the mean, with standard deviations of 1.06% and 0.79% for Day 1 and Day 7, respectively. The stability of Brownie Bears’ user retention makes now the perfect opportunity to introduce a change to potentially boost the rates.

As you know, user retention is one of the most important metrics in the free-to-play mobile game industry as it impacts both audience growth and revenue. Download metrics do not translate accurately to audience size if users stop playing soon after installation, while the likelihood users would spend money on our game via in-app purchases increases with the longer they play.

The simple change to increase user retention? Move the first timer gate from its current level to a higher level. The timer gates, as we refer to them, are the points in the game in which we insert wait timers to prevent users from immediately proceeding to the next level. Players can proceed to the next level after the timer has expired or by making in-app purchases to bypass the timer.

Hypothesis

Our first hypothesis is that by moving the first timer gate, we affect the user retention rate.

We further hypothesize that moving the first timer gate from level 30 to level 40 will improve Day 7 user retention rate by 2%, moving the current rate of 18–19% beyond this year’s goal of 20%.

Figure 3: Brownie Bears Retention Rate w/Day 7 Projections

Experiment Details

Implementation

For each new download during the experimental period, users will have an equal chance of downloading either the original version of the game with the timer gate at level 30 or the test version of the game with the timer gate moved to level 40. Internal simulations will be executed to ensure the installation rate for each version holds true at approximately 50% and the timer gate for the test version has been moved to level 40.

Duration

The experiment will execute for a period of 30 days. At the end of the 30 days, immediately disable the test version of the game as data analysis is executed. The simulation period will start two weeks prior to the experiment and run for 7 days. The 7 days in between will be used to make any adjustments to the implementation.

Data Collection

• Day 1 user retention rate
• Day 7 user retention rate
• User IDs
• Game version (control=gate_30, test=gate_40)
• Number of rounds played

Data Analysis

At the end of the experimental period, the Day 7 user retention rate for each game version group (gate_30 and gate_40) will be compared.

Benchmark(s)

Day 7 user retention rate will be the metric used to determine the next steps.

• If rate increases by 2% or more, we recommend making the timer gate change permanent.
• If rate does not significantly change or increases by less than 2%, we recommend re-executing the experiment for another 30 days and evaluate the cumulative data.
• If Day 7 rate decreases, make no change to the timer gate and end the experiment.

Secondary Metrics. We will also compare any differences in user retention for Day1 as well as the number of rounds played by each user.

Results Analysis

Now that the 30-day experimental period is over, let’s take a look at the data collected.

Version Distribution

To confirm that each version of the game was distributed equally to users, we counted the number of unique users for each.

Confirming ~50/50 split in game version distribution to users.

The data shows a near 50/50 split in game version distribution as intended:

• 49.56% for the original version of the game
• 50.44% for the test version of the game

Success Metric: Day 7 Retention

Since retention is recorded as TRUE (1) or FALSE (0), we can calculate retention by determining the ratio of number of TRUE values to the total number of values for the Day 7 retention field.

This is equivalent to calculating the mean as shown below:

Calculating Day 7 retention rates for each version of the game.

The data shows a decrease of 0.82% in Day 7 Retention Rate by moving the first timer gate from level 30 (Control) to level 40 (Test).

With a p-value < 0.05, we are confident the difference in retention rates is significant and not due to any noise in the data.

Calculating t-value and p-value for Day 7 retention rates.

Lastly, we use bootstrapping to give confidence to the calculated difference between the control and test Day 7 retention rates.

Calculating probability control Day 7 retention rate > test Day 7 retention rate.

Probability of control Day 7 retention rate > test Day 7 retention rate at several sampling rates.

Even at just 10% sampling rate (~9000 users), the probability the original version of the game will have greater Day 7 retention rate is 85%. This probability only increases as we move closer to 100% sampling. The average decrease in retention rate holds steady at 0.81%–0.84%, depending on the sampling rate used.

Secondary Metrics

• The data shows a decrease of 0.59% in Day 1 Retention Rate.

Figure 6: Day 1 Retention Rates and Delta

• The data shows nearly identical results with respect to the number of rounds played. Data was filtered first by excluding extreme outliers (50, 000 rounds!) and then again by excluding IQR outliers.

Figure 7: Rounds played statistics

Figure 8: Rounds played distribution, split by game version.

Conclusion and Recommended Steps

The experiment results tells us there is indeed a difference made in Day 7 retention by moving the first timer gate from level 30 to level 40. The t-test shows this difference is real and not due to data noise.

Additionally the bootstrap analysis shows high probability that moving the timer gate will actually decrease Day 7 retention rate, albeit slightly.

Given these two findings, we recommend making no changes to the placement of the first timer gate.

Link to analysis scripts.

Note: Analysis code can be found on GitHub.