Data Scientist 101: T-test or Z-test

Why is it so important to choose from T-test or Z-test in the A/B testing world? Wouldn’t you ever be so confused by this basic concept?

In statistical tests, the terms “large” and “small” sample sizes often relate to the central limit theorem and the distribution of the test statistic under the null hypothesis.

For the Z-test and T-test:

• A Z-test is typically used when the sample size is large. Traditionally, statisticians consider a sample size “large” if it is 30 or greater. This is because, with larger sample sizes, the sampling distribution of the sample mean becomes approximately normal due to the central limit theorem, regardless of the distribution of the population. So, if you have sample sizes of 30 or more, and you know the population standard deviation, you could use a Z-test.
• A T-test is generally used for smaller sample sizes (less than 30), or when the population standard deviation is unknown. The T-test accounts for the extra uncertainty by using a distribution that has “fatter tails” than the normal distribution. This distribution changes depending on the sample size, which is reflected in the degrees of freedom.

Here’s how you might consider whether to use a Z-test or T-test in your specific scenario:

Population Standard Deviation Known:

If, by “population,” we refer to the behavior of all similar email campaigns in the past, and your company has a lot of historical data on CTR for emails, you might have a reliable estimate of the population standard deviation. In this case, you could use a Z-test because:

• The sample size is large (the central limit theorem assures a normal distribution of the sample mean).
• You have a good estimate of the population standard deviation.

Population Standard Deviation Unknown:

More commonly, the exact population standard deviation is unknown and is estimated from the sample data. Even with a large sample size, a T-test is typically used because:

• The T-test adjusts for the uncertainty in the estimate of the population standard deviation.
• With large samples, the T-distribution (used in the T-test) is very similar to the normal distribution (used in the Z-test), so the results would be nearly identical.

In the case of A/B testing CTR for email campaigns, you’d likely use a T-test. Here are the steps you’d take:

1. Calculate the CTR for both groups: Determine the CTR by dividing the number of clicks by the number of emails sent (or opened, depending on how you define CTR) for each group.
2. Calculate the standard deviation for both groups: You’ll need the standard deviation of the CTR for both groups to determine the variability within each group.
3. Perform a T-test: Compare the two CTRs using a T-test to see if the new email copy significantly changed the CTR.

Practical Implications:

When interpreting your results, consider both statistical significance and practical significance:

• Statistical Significance: With a large sample size, even a small difference in CTR might be statistically significant. This could lead you to conclude that the new email copy affects the CTR.
• Practical Significance: Even if the result is statistically significant, you should consider if the difference in CTR is meaningful for the business. For example, if the new email copy increases CTR from 2.00% to 2.01%, that might not be a meaningful increase despite being statistically significant.

You should also consider the cost of implementing the new email copy against the expected increase in revenue from the improved CTR. If the cost outweighs the benefit, the change might not be practically significant.

So, in summary, for large sample sizes and when the population standard deviation is unknown (which is usual in A/B testing), you’d typically use a T-test. And when analyzing your results, always consider both statistical and practical significance.