A Guide to Calculating Sample Size and Experiment Length with Minimum Detectable Effect (MDE)
In the fast-paced world of technology, staying ahead of the competition requires continuous innovation and informed decision-making. Tech companies have realized the immense potential of experimentation as a powerful tool to validate ideas, measure the impact of changes, and make data-driven decisions.
AB testing is a technique used in digital marketing and product development to compare multiple variations of a webpage or app screen. By dividing the audience into different groups and measuring user behavior, organizations can determine which variation performs better.
In the world of experimentation (in this case is A/B Testing), determining the appropriate sample size and experiment length is crucial for obtaining reliable and statistically significant results. One key factor in this process is the Minimum Detectable Effect (MDE). In this article, we will explore how to calculate the sample size and experiment length based on the desired MDE, ensuring accurate and meaningful experimentation.
Understanding Minimum Detectable Effect (MDE)
The Minimum Detectable Effect represents the smallest change or difference in your experiment metric that you want to detect as statistically significant. It is essential to define the MDE upfront to ensure your experiment is adequately powered to detect meaningful effects.
Sample Size Calculation
To calculate the required sample size, you need to consider several factors:
a. Confidence Level: Choose a confidence level that represents your desired level of certainty, typically set at 95%.
b. Power: Determine the power of the experiment, which represents the probability of detecting an effect if it truly exists. A commonly used value is 80%.
c. Baseline Conversion Rate: Determine the current conversion rate or performance metric of your control group.
d. MDE: Define the desired Minimum Detectable Effect, which is the smallest change you want to detect.
Experiment Length Calculation
Once you have determined the sample size, you can estimate the experiment length. Consider the following factors:
Expected Traffic or User Volume: Assess the expected number of visitors or users during the experiment period.
Conversion Rate Stability: Evaluate the stability of your conversion rate over time.
Sample Allocation: Determine the percentage of users allocated to each variant in your experiment.
Statistical Significance Criteria: Choose the statistical significance threshold to determine when the experiment can be concluded.
Online Tools and Calculators
Utilize online sample size calculators and statistical tools designed specifically for A/B testing and experimentation. These tools simplify the calculation process by incorporating the necessary statistical formulas and considerations.
Real Case
When inputting a baseline conversion rate of 10%, a relative change of 5%, a power of 80%, and an alpha (significance level) of 5%, the resulting sample size needed per variant is 350k. This information helps determine the appropriate sample size for an AB test, ensuring statistical validity and reliable results.
If we have a total of 3.5 million users in a week, we actually only need 20% of all users (divided into two groups) to obtain a sufficient sample size with a 2% minimum detectable effect (MDE). However, if our active users in a week are less than 350k, we would need to extend the duration of the experiment week.
Conclusion: Accurate calculation of sample size and experiment length based on the Minimum Detectable Effect is crucial for successful experimentation. By understanding the concepts discussed in this article and leveraging available tools, you can ensure your experiments are properly powered and yield meaningful results. Conducting well-designed experiments will enhance your decision-making process and drive data-driven insights in your organization.
Remember, experimentation is an iterative process, and continuously refining your approach based on learning and outcomes will lead to more effective experiments and improved business outcomes.
