Experimental Design & How to avoid blowing everything up

There is a lot of finer detail into setting up, tracking, and interpreting the results of experiments. Understanding how to set up good experiments is a bit more than just setting up an A and a B variant and looking at what can impact these metrics. A little sense of scientific rigor is necessary to come to the right conclusions. Having an understanding of who should be in your experiment, how to measure its success and how to avoid the different pitfalls of experimenting using A/B tests is necessary to set up a good experiment.

User Assignment and population

A large part in setting up an experiment involves understanding who is going to be the population you are going to treat, how are you going to split the different control/variants groups and how far down the funnel are you going to measure user’s action with respect to the experiment setup.

Understanding who should be exposed to your experiment should be something that you have obtained through prior data, or through the generation of hypothesis for your experiment. You should have from your available data already pre-defined eligibility rules or specific segments that you want to run your experiment against.