I recently came across this book written by Ron Kohavi, Diane Tang, and Ya Xu — Trustworthy Online Controlled Experiments and have found it helpful with my A/B testing studies. I’d like to share my notes from this book.
Outline:
Chapter 2: Running and Analyzing Experiments: An End-to-End Example
Chapter 3: Twyman’s Law and Experimentation Trustworthiness
Chapter 4: Experimentation Platform and Culture
Chapter 1: Intro
One accurate measurement is worth more than a thousand expert opinions. — Grace Hopper
1.2 Why Experiment? Correlations, Causality, and Trustworthiness
1.3 Necessary Ingredients for Running Useful Controlled Experiments
1.1 Terminology
(1) Overall Evaluation Criterion (OEC): A quantitative measure of the experiment‘s objective.
- For example, your OEC might be active days per user
- The OEC must be measurable in the short term (the duration of an experiment) yet believed to causally drive long-term strategic objectives
- In the case of a search engine, the OEC can be a combination of usage (e.g., sessions-per-user), relevance (e.g., successful sessions…
