Hypothesis Testing Part III: How to determine the right significance level?

Significance level plays a pivotal role in hypothesis testing. Determination of the same is more driven by the problem you are trying to address rather than mere mathematical calculation. Let’s find out more about this small but very powerful concept through simple math.

Towards the end of the previous article, we have found out that the significance level is kind of a tolerance level for the hypothesis you are testing. If your p-value goes below this benchmark then you reject the null hypothesis else you fail to reject the same. But the point is how to determine the same? Well, it depends on your priority — whether you want to reduce type I error or type II error.

Don’t look so frightened as yet!!! Let’s find out about these two unknown terms through some examples.

Type I vs Type II error

Let’s meet Chetan, a business consultant, food enthusiast and a budding entrepreneur who is currently looking to start his own food truck business which specializes in healthy evening snacks. He is currently surveying people in the Enigma Tech Village(ETV), the same one he works at, to test demand for his products on offer before he applies for necessary permits to operate and invest in setting up the business. Since his product offerings are relatively new, he has set his

Null hypothesis( Ho): Demand is not high enough

Alternative hypothesis(Ha): Demand is high enough

Based on the hypothesis testing at 1% significance level(α) conducted on the samples of the survey responses, he is kind of convinced that this business is not going to run well as there is not enough demand. So, he dropped the idea. Basically, he did not/failed to reject the null hypothesis. A few months later, he observes that someone with the same concept has opened a shop and the shop is running extremely well. Chetan is now heartbroken.

So here, Chetan has done a Type II error — which is defined as failing to reject the null hypothesis while the null hypothesis is false.That means, Chetan thought that demand would not be high enough and he did not start the business (failed to reject the null hypothesis) whereas the demand was high enough(Null hypothesis was infact false).

This is an extremely tough time for Chetan but looking at the success of his potential competitor he again starts collecting surveys from another tech park called Vigman Tech Park (VTP) and this time he thought to himself even if his collected samples show 10% chances of high enough demand(significance level at 10%) he will start the business. So, he has conducted the hypothesis testing and rejected the null hypothesis based on the result. Or, in other words, Chetan now has his own food truck business. Few months have gone by but the business is not doing good at all as expected and soon he has to shut shop and in the whole process, he suffers a huge loss.

In this case, Chetan has done a Type I error — which is defined as rejecting the null hypothesis while the null hypothesis is true.That means,Chetan thought that demand would be high enough and he started the business (rejected the null hypothesis) whereas demand was not high enough(Null hypothesis was infact true).

Later Chetan realizes that most of the companies at VTP have their own cafeteria which is not the case at ETV- which in a way contributed towards his failed venture.

The relationship among significance level, type I and type II error

Now let’s analyze both the situation Chetan was in and significance level(α) he set up in each of the cases:

• In the first situation(type II error), he was stringent and set up a very low α and he only lost a missed opportunity.
• In the second scenario(type I error), he became more relaxed by setting up a higher α and he actually burned the cash.

If he would have kept the significance level as low as before then probably could have avoided the misfortune. So, in order to avoid losing money, Chetan should have concentrated on reducing the chances of committing type I error and hence kept α as low as possible.

Let’s take another example where you will be required to keep type II error under control rather than type I.

Let’s meet Mr. Rohan Kumar who happens to be the secretary of the Spectra Sylvian Society. For the past few days, he has received a lot of complaints about the chlorine level in the society’s swimming pool. So, he has instructed a team to test the water. He has decided that he will close the pool temporarily if the chlorine level exceeds the acceptable limit. So, here

Null Hypothesis(Ho): The chlorine level is acceptable

Alternative Hypothesis(Ha): The chlorine level is not acceptable

Like Chetan, Mr.Rohan has also two choices of α — keep it low or keep it high. If Mr. Rohan chooses to be stringent in selecting the α(let’s say 5% in this case) and closes the pool despite the fact that the chlorine level is acceptable then he will commit a type I error. On the other hand, if he stays firm in the selection of α and decides not to close the pool despite the fact that the chlorine level is beyond the acceptable limit and he will end up committing a type II error which is more dangerous in this setting in terms of the health and safety of the inhabitants. Let’s p-value in this setting is 9%. So, even if the p-value has not hit the significance level(5%), 9% chances can still lead to greater health and safety concerns. But if he chooses 10% instead of 5% to be the α then he will close the pool and that will be good for society. So here, it will be a good idea to set a high value of α to reduce the chances of committing a type II error.

So, in a nutshell,

• if your priority is to reduce type I error keep a low α
• If your priority is to reduce type II error keep a high α