Vijai K Rangan

… heavy soda consumers between the two groups may have a disproportionate impact on our conclusions. When running online experiments, consumer internet products often face similar issues related to their most active users, whether it is in measuring a change to a metric like streaming hours at Netflix or perhaps messages sent or photos shared in a social app.

…ation, ranging from those who hardly consume any soda to those who consume copious amounts. Second, heavy soda consumers may represent a small percentage of the population, but they could account for a large percentage of overall soda consumption. Therefore, even a small imbalance in heavy soda consumers between the two groups may have a disproportionate impact on our conclusions. When running online experiments, consumer internet products often face similar issues related to t…

It is predictive of success in the second stage: the metrics measured in the first stage are aligned with our core A/B evaluation metrics.