Causation vs Correlation

One of the challenges while measuring effectiveness of new product or feature launch is to differentiate between causation and correlation of the launch and impact on the KPIs. Generally, the best option is to run as an experiment. Launch the new experience to randomized test user group and measure the KPIs of the test group against the control group, control group being the users in original experience. Experimentation or A/B testing statistical measurement methodologies are designed to identify causation or actual impact of new experiences. However, not all digital experiences qualify as candidates to run experiments on it. Some of the examples where running experiments during product launch is not feasible include the following:

• Low traffic pages/experiences
• Time pressure to do general availability release
• No practical way to randomly assign the end users
• Complex dev environment/experiment set-up
• Development resources constraints
• Lack of experiment tool in the org

In these cases, you may have to launch products without setting up experiments. Now, you may even see lift in conversion after the launch, but that just proved correlation. It does not necessarily prove that the new product launch drove the improvements. One of the prominent examples of correlation but not causation is ice cream sales and shark attacks.