p-Value
Hope you read my previous blog on Hypothesis Testing before continuing.
We ended the previous blog saying that we use p-Values to test the null hypothesis. Typically, we reject the Null Hypothesis if p < 0.05 (95% Confidence).
Why do we take p-Value of 0.05 as a threshold?
Remember that our null hypothesis is that Drugs A and B are same. So we are interested in the extremes on both ends. +/- 2 (or) 3 standard deviations and beyond. So we calculate the probability that the experiment sample falls in either extremes (2.5% + 2.5%)
Definition of p-Value: “Probability of getting values equal or more extreme than the experiment value, **given the null hypothesis is true**”
Other ways of interpreting p-value:
1. “How many % of experiments differ only by random things”.
If p=0.9, 90% of experiments differ only by random things. So there is no difference in the intervention.
2. “How many times are you willing for false positives?” (Type-1 Error)
If p=0.05, it can still mean that the sample belongs to the Null-Hypothesis (See the figure above). It’s extreme, but still possible. By rejecting the null hypothesis (that Drug A = B), I’m telling that I’m ready to accept 5% false positives. That means, 5% of the times, I may say Drugs A and B are different, even if they are the same.
Caveats:
- p-Value can be hacked. Many frauds do it to claim that they have done science. In the next blog, we’ll see how such fraud is done.
- p-Value only tells you about statistical significance. It can tell you that Drug A is different from Drug B. But how different? Only 1%? 5%? 50%? p-Value won’t tell about it. We’ll discuss these in next blogs.
