The Relationship Between Hypothesis Testing and Confidence Intervals

What Is Hypothesis Testing

Hypothesis testing is an act in statistics whereby an analyst tests an assumption regarding a population parameter. The methodology employed by the analyst depends on the nature of the data used and the reason for the analysis.

Hypothesis testing is used to assess the plausibility of a hypothesis by using sample data. Such data may come from a larger population, or from a data-generating process. The word “population” will be used for both of these cases in the following descriptions.

Hypothesis testing allows us to draw a conclusion on how plausible a certain hypothesis is using sample data from a population. That in reality, the relationship or effect we are seeing between two variables isn’t just due to pure luck or chance!

An example of a typical hypothesis test (two-tailed) where “p” is some parameter

we state our two kinds of hypothesis:

Null hypothesis (H0): The “status quo” or “known/accepted fact”. States that there is no statistical significance between two variables and is usually what we are looking to disprove.

Example: H0 = 0 ; There is no difference between heart rate before and after exercising.

&

Alternative Hypothesis (H1/Ha): The opposition of the null, and is what we are testing for statistical significance.

Example: H1 ≠ 0 ; There is a difference between heart rate before and after exercising.

Second, depending on the problem, we choose a test accordingly and from the result observe a test statistic.

Say our data follows a standard normal distribution, we use a z-test statistic, obtain a p-value, and from that, draw a conclusion.

Using this test statistic or p-value we can then compare this to our α of 0.05.

If smaller, we reject our null hypothesis and conclude with our alternative hypothesis. If larger, we fail to reject our null hypothesis and conclude with null hypothesis.

But hold on, we can also draw a conclusion from not only using p-values but also from using confidence intervals because of the relationship between CI and hypothesis tests!

What is a Confidence Interval (CI)?

A range of possible values that is likely to capture an unknown parameter, given a certain degree of probability (confidence).

Using this formula we can calculate a confidence interval!

Confidence Interval Formula by Google

Confidence Level % = 1 − α

Alpha (α) is known as the significance level or accepted error; an α = 0.05 is typically a good level of accepted risk, but varies depending on the situation.

So typically, you’ll see things like “95% CI” and a range of values like in the example table below.

This table above is from the first study that determines the actual heart age for U.S. adults from ages 30–74. The study can be found here.

Crazy how adults within the age 40–49 are likely to have an excess heart age that is roughly 6 years older with 95% confidence!

Esentially, we are saying if we were to sample many many times, and calculate confidence intervals for a certain parameter like a mean or regression coefficient, we can then expect about 95 out of 100 of those intervals to capture the true population parameter.

The Connection?

Confidence intervals and hypothesis testing share the characteristic that they are both inferential techniques which use a sample to either estimate a population parameter or test the strength and validity of a hypothesis.

This image here is a golden nugget that I think is tremendously helpful in better conceptualizing this relationship.

We see here that the point of reference is what is different. Hypothesis tests are centered around the null hypothesized parameter and confidence intervals are centered around the estimate of the sample parameter.

How can use these two concepts in tandem?

If the null hypothesized value is found in our confidence interval, then that would mean we have a bad confidence interval and our p-value would be high. Typically our null hypothesized value will be 0 (point of no difference), and if we find 0 in our confidence interval then that would mean we have a good chance of actually finding NO DIFFERENCE, which is typically the opposite of what we want.

In other words, if the null hypothesized value falls within the confidence interval, then the p-value is always going to be larger than 5%. Conversely, if the null hypothesized value falls outside of our confidence interval then the p-value is going to be less than 5%.

Some Takeaways

• Well for one we know, if we use an α of 5% in our hypothesis test conversely, we will also be using a 95% confidence interval since alpha levels and confidence intervals always correspond with each other.
• Confidence intervals and hypothesis testing are both methods that look to infer some kind of population parameter from a sample of data drawn from that population.
• Confidence intervals gives us a range of possible values and an estimate of the precision for our parameter value.
• Hypothesis tests tells us how confident we are in drawing conclusions about the population parameter from our sample.
• When 0 is included in our confidence interval this means we are likely seeing that there is no difference between our sample and the population parameter. Additionally, the p-value from our hypothesis test is probably higher than our alpha and we‘ll likely fail to reject our null hypothesis!
• Both confidence intervals and hypothesis intervals can be used in tandem to help support our conclusions!