Bayesian A/B Testing with Continuous Variables — including Python Code

This article discusses how Bayesian estimation can be implemented in your digital experimentation methodology, with a specific focus on computation for continuous, non-discrete metrics

Figure created by Author (Frank Hopkins, 2020)

When an experiment on your site has been paused, there is a chance that not all the information needed to either accept or reject your hypotheses has been captured in your MVT platform. If this is the case you will likely have to perform some form of post-hoc significance testing on raw experimentation, user-level data.

If you are working with a continuous metric — which are variables that can take on an unlimited range of values between both the lowest and highest points of a given measurement — the T-test is probably most familiar to you. However, the T-test and more broadly speaking frequentist methods in general, can be flawed in their practical applicability. Firstly, frequentist methods rely on pre-determined power analyses to calculate required sample sizes and subsequently the desired length of time an experiment should run before comparing differences between groups. The problem with this form of inference is it fails to work in a dynamic or ad-hoc setting and if sample size requirements…