Image by author

Bayesian Experimentation Metrics Explained

At Typeform we’ve adopted GrowthBook as our experimentation platform for AB testing. GrowthBook calculates 3 Bayesian statistics on your metrics to help you draw conclusions about your new features:

• Uplift
• Chance to Beat Control
• Loss or Risk of Choosing Variant

In this article I’ll explain what they are and how to interpret them.

image source

Coin Flip Example

To help explain our statistics we’re going to run a thought experiment using a classic coin flip example — probably the best way to explain Bayesian statistics.

Suppose you have 2 coins:

• Coin 1 (control) is a fair coin with a 50:50 chance of heads or tails
• Coin 2 (variant) is unknown and you’d like to know whether it has a higher chance of giving a heads

We’re going to experiment to test the bias in coin 2 against coin 1.

Experiment Outcomes

We’ll run through 3 different versions of the experiment flipping the coins a different number of times in each case: