Multiple Testing

How Can We Test Multiple Hypotheses?

Alex Harlan
Sep 4, 2018 · 2 min read

Let’s say we have a set of hypotheses that we want to test at the same time. Our first thought might be to test each hypothesis separately, using some level of significance α. Sounds like a decent enough idea.

But let’s consider a case where we have 15 hypotheses to test, and a significance level of 0.05. What’s the probability of observing at least one significant result just due to chance?

P(at least one significant result) = 1 − P(no significant results) = 1 − (1 − 0.05)**15 ≈ 0.53.

So, with 15 tests being considered, we have a 53% chance of observing at least one significant result, even if all of the tests are not actually significant. That’s going to be a problem if we have many hypotheses to test. So how can we test multiple hypotheses without increasing our probability of observing a significant result just due to chance?

Bonferroni Correction

The Bonferroni correction is a method for correcting for this phenomenon. The significance cut-off at α/n where n is the number of tests. In our previous example, with 15 tests and α = 0.05, you’d only reject a null hypothesis if the p-value is less than 0.003333. Now if we calculate the chance of observing a significant result by chance we get, P(at least one significant result) = 1 − P(no significant results) = 1 − (1 − 0.003333)**15 ≈ 0.04885. This is much closer to our desired level of .05, it’s even a bit under so we are being conservative here.

P-Hacking

Failing to use the Bonferroni correction is a type of p-hacking.

P-hacking is the conscious or subconscious manipulation of data in a way that produces a desired p-value, typically in the form of obtaining a significant result that is not actually significant. Assuming that we are honest researchers we want to avoid p-hacking when we are performing analysis so that we don’t come to erroneous conclusions. As the saying goes, torture your data long enough and it will confess.

Alex Harlan

Written by

A used to be math major, doing things with data. Data Analyst @ FairTradeUSA alexforrest.github.io

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade