Member-only story

Why we moved away from Conversion Rate as a primary metric

The impact of violating assumptions of independence on AB Tests and changing the primary metric for your company

Bhavik Patel
TDS Archive
7 min readJun 24, 2021

--

Photo by Jonathan Petersson on Unsplash

Changing the primary metric for any company is not a straightforward thing to do. Why you may ask? There are dozens of reasons but here are some of the ones that come to my mind:

  • The company has been using that metric for a long time
  • Forecasts have been made from historical values
  • Team OKRs include targets against it
  • People understand the metric and the levers that drive it
  • Deviations from the expected values can easily be spotted
  • Causes for change to the metric can be quickly identified

This shortlist alone would cause anyone to break out into a sweat! So imagine my surprise when in my first month at Gousto, the Head of Product for our Menu Tribe suggested that our primary metric, Menu Conversion Rate (MCVR) was likely to be the wrong metric for us to be using for our experiments as it violated assumptions of statistical independence.

Violated the who with the what now???

--

--

TDS Archive
TDS Archive

Published in TDS Archive

An archive of data science, data analytics, data engineering, machine learning, and artificial intelligence writing from the former Towards Data Science Medium publication.

Bhavik Patel
Bhavik Patel

Written by Bhavik Patel

Director of Experimentation and Analytics. Ex-Director/Head of Analytics at Hopin/Gousto/MOO/PhotoBox. Twitter: @dodonerd

Responses (2)