Detecting Sample Ratio Mismatch in your A/B Tests

This post helps you identify whether you have unbalanced cohorts in your digital experiments and incorporates some basic Python functions to aid in SRM detection

Image created by Author (Frank Hopkins)

Introduction

When an experiment is paused and we look to investigate the statistics of a given test we often ponder on our findings. We may have failed to detect an overall statistically significant result and therefore stratify our data by age, device, geo-location or platform type to see if our experimental condition performs preferentially for a given segment. Furthermore, we may adjust the significance threshold in post-hoc testing to see if we reached statistical significance for a variety of alpha thresholds, in order to provide stakeholders with a level of confidence in a given experimental condition. Both of these additional pieces of analysis are perfectly valid methods for assessing your confidence in your results, but could be invalidated if the data you are collecting lacks basic integrity. It is therefore essential that prior to making causal claims about our data that we perform statistical tests during our experiments to ensure the data we are collecting is within our expectations of how experimentation data should be collected.