A Comprehensive Understanding of Bayesian A/B Testing

Minimal Theory to Understand and Use Bayesian A/B Testing

I have reviewed a few blogs online in hopes of understanding Bayesian A/B testing, but found them either too advanced, jumping to some conclusions abruptly, or too simplistic, either way, are not giving me enough confidence to actually understand and apply it.

In this post, I will try to give you a step-by-step introduction to Bayesian A/B testing with theory explained (which won’t be as hard) and some techniques and trade-offs that you need to heed when applying it.

Table of Contents

• The Frequentist Dilemma
• Define the Problem
• Some Prerequisites
  - Probability distribution
  - Prior distribution
  - Posterior distribution
• Answer the Question
• The Loss Function
• Conclusions
• Credits and References

Photo by Edge2Edge Media on Unsplash

The Frequentist Dilemma

Well, it’s an exaggerated phrase. It is not actually a dilemma, but the traditional A/B testing of frequentists does fail to answer a most caring question,

What is the probability that variant B is…