(Source)

The Problem with P-values

Statistics and the Replication Crisis

If you have ever taken a statistics course or read a scientific paper, then you have probably seen the phrase “p-value.” The ubiquitous statistical tool is just about everywhere in modern science. It’s an extremely convenient way to establish that a result is noteworthy and important.

In case you need a refresher, let’s go over what a p-value actually is. To get one, we first need to conduct a hypothesis test. Once you have your data, you will typically have a “hunch” about it. You want to prove that this hunch is true, but you have to account for the fact that it might not be. This is like the “innocent until proven guilty” concept in the law.

There are a lot of nuances to defining these two sets of statements, but you want to cover all possibilities between the two of them.

Next, we perform some kind of test. We are trying to calculate the probability that we got the sample we did (or a more extreme one) if the Null Hypothesis were true. I won’t go into the details here, but statistics is full of different methods for doing this. The outcome is a p-value. So far, this discussion has been…