How to Analyze A/B Test Results?

A/B Test Diagram from https://www.vippng.com/maxp/hxxximo/

In this article, I will take you through analysing a hypothetical A/B Test conducted by a mobile game developer.

Game Overview

• The game is called Math Puzzle Game. Like other puzzles, players need to beat levels to progress.
• The game is Free to Play meaning that players can download and play the game for free but can optionally buy hints on the levels.

Description of Experiment

• An experiment (A/B Test) was run on the game, offering two different game experiences that we call A and B, group A being the control group where the experience is kept as is, and group B being the experiment group that is exposed to the new experience.
• We set the assignment process to randomly distribute players among the groups: 80% to group A (control) and 20% to group B (test).
• The experiment ran from 2017–05–04 to 2017–05–22.

Experiment Hypothesis

• The hypothesis was that the new change to the game would improve the user experience making Math Puzzle Game a better game.

Data

We have the 2 tables

1. Assignment table: