Steps to A/B Testing using MS Excel

A/B test experiment is basically an experiment with two groups A(control) & B (treatment) with typically a hypothesis that treatment is better than control. A/B tests are common in web designing and marketing for example publishers like NYT A/B test their web headlines to determine which produces more clicks.

Switching back to the process, ideally, subjects are randomly assigned or randomized to the treatment group so that it can be concluded that the difference between A & B is either due to the effect of treatment or luck of the draw where incidentally naturally better-performing subjects were placed in A or B.

An interesting question would be to ask, why have a control group, why not compare the results with prior experience? It is so that while comparing the two outcomes, you’re able to keep “everything else” to be the same. In case of comparing it with prior experience, there is a chance that “other things” might be different. For example, comparing number of clicks on a website for an experiment that ran on Monday with previous day’s data could potentially lead to wrong interpretations. It is because assuming “everything else” to be same on a weekday and a weekend doesn’t sound right.

Why not A/B/C/D? This type of experiment actually saves time and since it is performed with more than two groups it is called a multi-arm bandit.

Choosing a test statistic

A test statistic or a metric is used to compare group A to group B. Perhaps the most…