A/B Testing: Materials for In-Depth Study
Recently, I got involved in the adventure of A/B testing. I decided to go beyond simple math and online calculators and dig deeper into this field. In this journey, I found many useful articles, papers, researches and so on. If you, like me, want to become professional in A/B testing, then check the following links in this post. You’ll find many exciting things.
Overview of the A/B testing field
These links are great to refresh your knowledge, remember the basic formulas, and to learn common mistakes and solutions for them.
- Guidelines For Ab Testing — Hooked on Data
- Controlled Experiments on the Web: Survey and Practical Guide
- Seven Rules of Thumb for Web Site Experimenters
Advanced Topics
Sequential Analysis
- Simple Sequential A/B Testing — Evan Miller
- Rapid A/B-testing with Sequential Analysis | Audun M Øygard
- Estimation in Sequential Analysis | Audun M Øygard
Peeking Problem
- How Not To Run an A/B Test — Evan Miller with wrong conclusions and great response to it A/B Testing Rigorously (without losing your job) (and extension A/B Testing With Limited Data)
- The Fatal Flaw of A/B Tests: Peeking | Lucidchart Blog
- Peeking at A/B tests: continuous monitoring without pain | the morning paper
Bayesian
- Bayesian vs Frequentist A/B Testing (and Does it Even Matter?)
- Discussion on Reddit — Frequentist or Bayesian AB Testing Methodology? : statistics
- Bayesian A/B Testing at VWO
Bayesian vs. Peeking Problem
- Is Bayesian A/B Testing Immune to Peeking? Not Exactly — Variance Explained
- Bayesian AB Testing is Not Immune to Optional Stopping Issues | Analytics-Toolkit.com
A/A
More
In addition to authors that wrote previous papers and articles, I recommend you to check out these resources:
- Ronny Kohavi is a Microsoft Technical Fellow and Vice President of Analysis & Experimentation. You need to explore his project ExP Platform. Also look for his recommendations on what to read.
- Eytan Bakshy’s blog. Eytan is a senior scientist on the Facebook Core Data Science Team, who lead the Adaptive Experimentation group.
- Evan Miller is a developer of statistical software. He has a series of A/B testing articles on his website.
- Papers from the SIGKDD conferences. It’s a community for data mining, data science, and analytics.
- The story “How Optimizely (Almost) Got Me Fired” and a paper about The New Stats Engine, where they fix the problems.
- Companies Blogs:
- Uber — Under the Hood of Uber’s Experimentation Platform
- Netflix — Quasi Experimentation at Netflix — Netflix TechBlog — Medium
- Airbnb — Scaling Airbnb’s Experimentation Platform — Airbnb Engineering & Data Science — Medium
- Booking — How Booking.com increases the power of online experiments with CUPED
- Facebook — Efficient tuning of online systems using Bayesian optimization — Facebook Research
- Quora – A Robust Statistical Test for Ratio Metrics
- Yandex – Online Evaluation for Effective Web Service Development
P.S. This article will be updated periodically.