The ultimate guide to A/B testing. Part 1: experiment design

A/B testing is a very popular technique for checking granular changes in a product without mistakenly taking into account changes that were caused by outside factors. In this series of articles, I will try to give an easy hands-on manual on how to design, run and estimate results of a/b tests, so you are ready to go and get these amazing statistically significant results!

In the first article, we will talk about approaches that are the most suitable for designing experiments and estimating the number of users per group needed to be sure in results.

Let’s say you have a game and are trying to increase retention rates by adding an additional gameplay mode. So, you spend some time and effort adding this new mode. At the end, you see that 10% more people have started to churn after the first day. Looks like these changes weren’t well received! And now investors are baying for blood. Word on the street is they are starting to wonder if is this the right time to give the old heave-ho to the CEO? Or to the product manager? Or maybe to the artist (they all disliked this new color on the menu button anyway)? But at the same time, your game has been reviewed by a very popular YouTube blogger. This has helped you get an additional 500k installs just in 2 days. However, the overall quality of this traffic is supposed to be much…