Bayesian Analysis for A/B Testing

Applying Bayesian methods to the difference of two proportions

A common use of statistics in business is the A/B test. This type of test pits two versions of the same experience — like a website page — against one another. The research question is simple to state: which version is better?

There are a panoply of tools and guides to assist this type of testing. This article looks at Bayesian analysis for A/B testing.

The analytical core

What is a probability? That question forges a fundamental schism in statistics.

Bayesian statisticians interpret probabilities as the degree of belief about events. Our prior belief — based on almost all coins being fair — is the probability of Heads is 1/2. After flipping the coin, we update our belief with Bayes’ Theorem.

Rev Thomas Bayes wrote the theorem in 1763. (Photo: Matt Buck/Flickr)

Suppose our interest is the arithmetic difference in conversion. As alternatives, we could choose the relative difference (‘lift’) or odds ratio.

The Bayesian framework says: the difference is a random variable. Bayesian analysis has the same core:

• Prior distribution: state our prior beliefs as a probability distribution;
• Receive data: we collect data, relating to our unknown…