Inferential Statistics — Hypothesis Testing

Logic of hypothesis testing and types of errors

Anjana K V
Nov 24, 2020 · 6 min read
Image for post
Image for post
Photo by Kevin Ku on Unsplash

You might have come across innumerable claims and statements involving numbers, especially in marketing campaigns and ads. “9 out of 10 doctors recommend Colgate toothpaste”, or “Dettol kills 99.9% of bacteria” are classic examples of numerical claims. The statistical validity of such statements regarding a parameter can be tested if you collect some sample data and perform certain calculations on it. This is the foundation of inferential statistics using hypothesis testing.

Image for post
Image for post
Few popular claims involving numbers

Hypothesis testing is a statistical testing mechanism to evaluate the evidence in the form of data to validate the hypothesis or assumption regarding a parameter. You start with two hypotheses — null and alternate, taking the two opposing sides to explain the parameter.

Image for post
Image for post
Logic of Hypothesis Testing

For example, if your car manufacturer claims that the car gives a mileage of 25 km per liter, first collect the mileage of multiple cars of the same model. This will form your sample data. Then, set the null and alternate hypotheses. Perform the test calculations and based on the results, evaluate the validity of the claim.

Null Hypothesis — Default position or the as-is state where the claim about a parameter is accepted as the truth

Alternate Hypothesis — Opposite of null hypothesis where the claim in null is contradicted and where the observed data is considered as the truth

Hypothesis testing is performed in order to either reject or not reject the null hypothesis based on the calculations done on the observed (collected) data. If the test rejects the null, then automatically one accepts the alternate position. On the other hand, if the test fails to reject the null, then we continue with the default position as status quo until we collect further evidence.

Let’s consider the following problem statement.

An incandescent light bulb manufacturer claims that the mean lifetime of a bulb is 2,000 hours or more. In a sample of 30 bulbs, it was found that they only last 1,900 hours on average. The sample standard deviation is 150 hours. At .05 significance level, can we reject the claim by the manufacturer?

Now, let us formulate the null and alternate hypothesis for this scenario.

Ho : Average lifetime of the bulb ≥ 2000 hours(claim)

Ha : Average lifetime of the bulb < 2000 hours (as per the sample data)

The next step is to perform the statistical test to assess whether we have to reject or not reject the null or the as-is state based on the data. Through the means of test calculations, we evaluate whether the evidence from the data is strong enough to disprove the null state and accept the alternate position. If there is no strong evidence, we continue to accept the claim.

The t-score is a test statistic for t-tests that measures the difference between an observed sample statistic and its hypothesized population parameter in units of standard error. A t-test compares the observed t-value to a critical value on the t-distribution with (n-1) degrees of freedom to determine whether the difference between the estimated and hypothesized values of the population parameter is statistically significant.

Image for post
Image for post
t-distribution for different degrees of freedom (v)

In this case, we will perform a t-test and calculate the associated p-value to make a decision. The t-test statistic is calculated as (x-μ)/(s/√n), where x is the sample mean, μ is the hypothesized or the claimed mean, s is the sample standard deviation and n is the sample size.

In the above example, the t-test statistic value will come out to be -3.65. In order to make a decision at 0.05 significance level (corresponding to 95% confidence level), the threshold value based on the t-distribution is -1.699. This can easily be obtained from the t-distribution table or by a quick formula in excel as T.INV(significance value [usually 0.05 or 0.1], sample size — 1).

The final step is to compare the test-statistic against the cut off value. If the abs(test statistic) > abs(cut-off value), then we reject the null hypothesis, else we do not. The purpose of the cut-off value is to determine how large a deviation from the claimed value (null hypothesis) will be sufficient to reasonably reject the null state. Typical industry standards are 95% and 90% significance levels, indicating a probability of error of 5% and 10% respectively. In a distribution with the hypothesized value as the average point, if the observed value from data is sufficiently far away from claimed value, then it will mean that it is unlikely to observe such data if the null-state was to be true. This forms the basis for the rejection of null. If the observed value is not too far from the claimed value, then we do not reject the claim.

Image for post
Image for post
P-value concept

In our example, the absolute value of test statistic is 3.65 which is greater than the cut-off value of 1.699. Hence, we reject the claim by the manufacturer that the average lifetime of a bulb is 2000 hours or more.

Here’s a quick guide to choose the type of test to perform based on the information available at your disposal.

Image for post
Image for post
Choice of Hypothesis Test

There are two types of errors associated with hypothesis testing — Type 1 and Type 2 errors.

The type 1 error occurs if we incorrectly reject the null hypothesis when in reality it was not to be rejected. In the previous example, if the average lifetime of the bulb was indeed 2000 hours or more as claimed by the manufacturer, but the sample we collected contained more number of defective pieces than usual, then we would have committed a type 1 error. α is the probability of committing type 1 error, which is the significance level we choose for the test. If we go for a 0.05 significance level, it means that there is a 5% probability of type 1 error to happen in the test outcome.

The type 2 error occurs when we fail to reject the null hypothesis when in reality we should have. β is the probability of committing a type 2 error. This is also called the power of the hypothesis test, as this determines the ability of the test to assess the strength of evidence in data to reject the null state.

Image for post
Image for post
Confusion Matrix for Hypothesis Testing

The values and importance of the errors are determined by the nature, complexity and context of the problem you are solving. In the case of medical tests, it is more important to not commit type 2 error wherein an actual patient does not get diagnosed with the ailment. It is okay even if there are false positives, but false negatives pose a serious threat in this scenario. On the other hand, in matters of adjudication, it is necessary to attain a low type 1 error where an innocent person gets convicted. Although it is equally important to have low type 2 error (guilty being judged innocent), type 1 error gets more importance because of the judiciary policy of ‘innocent until proven guilty’.

Image for post
Image for post
Examples for Type 1 and Type 2 errors

Thus, depending on the context, the objective of the hypothesis testing and the importance of associated errors varies. Hypothesis testing is an integral part of inferential statistics, which forms the basis for a lot of advanced machine learning algorithms.

The Startup

Medium's largest active publication, followed by +756K people. Follow to join our community.

Anjana K V

Written by

Data Science Professional | 6+ years of experience in analytics across various domains — retail, insurance, finance and digital advertising

The Startup

Medium's largest active publication, followed by +756K people. Follow to join our community.

Anjana K V

Written by

Data Science Professional | 6+ years of experience in analytics across various domains — retail, insurance, finance and digital advertising

The Startup

Medium's largest active publication, followed by +756K people. Follow to join our community.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store