CUPED R-Shiny Tool

Introduction to CUPED

Recently I have been delving into ways in which experiment sensitivity can be increased — via reduction of variance associated with pre-experiment information. If a strong correlation can be determined between a pre-experiment metric (covariate) and experimental metric, the variance that exists within your experimental metric prior to conducting your experiment can be controlled for, prior to conducting any significance testing. There are a few different procedures worth considering when implementing controlled using pre-experiment data (CUPED) methods, but I have written about ‘covariate methods’ in previous Medium posts (adapted from Booking.com). Covariate methods focus on the relationship between a pre-experiment and experimental metric (this is typically a continuous variable), and omit variance that existed prior to your experiment starting. Reduction of said variance leads to an increase in experimental power, which means you are more likely to detect statistically significant effects (consider this a means of noise reduction in order to detect any true effect that may or may not exist between your experimental conditions).

My previous work on CUPED methods included an example using some of my work in Python and how pre-experiment information is merged to your experimentation data on cookie_id units. Sadly performing this sort of analysis can…