Introducing the Experiment Guardrails framework we implemented at Airbnb, which helps us prevent negative impact on key metrics while experimenting at scale.

by Tatiana Xifara, Reid Andersen & Ali Rauh

Image for post
Image for post

Each week, thousands of online experiments run concurrently on the Airbnb platform to measure the impact of potential product changes monitoring approximately tens of metrics per experiment. When making launch decisions, each team is often focused on different evaluation criteria — for example, the Trust team prioritizes Fraud Identification, while the Experiences team may prioritize discovery of the Online Experiences product in our Homepage. Experiments that positively impact one team’s…

Tatiana Xifara

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store