Introducing the Experiment Guardrails framework we implemented at Airbnb, which helps us prevent negative impact on key metrics while experimenting at scale.
by Tatiana Xifara, Reid Andersen & Ali Rauh
Each week, thousands of online experiments run concurrently on the Airbnb platform to measure the impact of potential product changes monitoring approximately tens of metrics per experiment. When making launch decisions, each team is often focused on different evaluation criteria — for example, the Trust team prioritizes Fraud Identification, while the Experiences team may prioritize discovery of the Online Experiences product in our Homepage. Experiments that positively impact one team’s…
