Evaluation of Online Controlled Experiments

Dec 6, 2019 · 1 min read

A/B Testing is the gold standard to estimate the causal relationship between a change in a product and its impact on key outcome measures. It is widely used in the industry to test changes ranging from simple copy change or UI change to more complex changes like using machine learning models to personalize user experience. The key aspect of A/B testing is evaluation of experiment results. Designing the right set of metrics — correct outcome measures, data quality indicators, guardrails that prevent harm to business, and a comprehensive set of supporting metrics to understand the “why” behind the key movements is the #1 challenge practitioners face when trying to scale their experimentation program. On the technical side, improving sensitivity of experiment metrics is a hard problem and an active research area, with large practical implications as more and more small and medium size businesses are trying to adopt A/B testing and suffer from insufficient power. In this tutorial we will discuss challenges, best practices, and pitfalls in evaluating experiment results, focusing on both lessons learned and practical guidelines as well as open research questions.


  • Somit Gupta, Microsoft.
  • Xiaolin Shi, Snap Inc.
  • Pavel Dmitriev, Outreach.
  • Xin Fu, Facebook.
  • Avijit Mukherjee, Facebook.

Tutorial @ WSDM 2020, February 3, 2020.

Here are the slides we used for the Tutorial. Feel free to post questions in the comments section and we will try to answer them.

Online Metrics Tutorial — Part I. Introduction.pptx

Online Metrics Tutorial — Part II. Best Practices.pptx

Online Metrics Tutorial — Part III. Advanced Topics.pptx

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade