What is meant by significance level ? Example with Python

https://vitalflux.com/wp-content/uploads/2021/12/level-of-significance-and-p-value.png

The significance level, also known as alpha, is the probability of making a type 1 error in a statistical hypothesis test. This means that it is the likelihood of rejecting the null hypothesis when it is actually true.

Imagine that you are trying to decide whether a coin is fair or not. You can flip the coin a few times and count the number of heads and tails to see if the proportion of heads is close to 0.5. If the proportion of heads is significantly different from 0.5, you can reject the null hypothesis that the coin is fair.

The significance level (alpha) is the probability that you will reject the null hypothesis even though the coin is actually fair. In other words, it is the probability that you will make a mistake by thinking that the coin is not fair when it actually is.

Here is a simple python code example to illustrate the concept of alpha:

import numpy as np
from scipy.stats import t

#Define the number of coin flips we will do
n = 10

#Flip the coin n times and count the number of heads
heads = np.random.binomial(n, 0.5)

#Calculate the proportion of heads
proportion_of_heads = heads / n

#Use the t-test to calculate the probability that the coin is fair (null hypothesis)
t_value, p_value = t.interval(alpha=0.05, df=n-1, loc=0.5, scale=0.5/np.sqrt(n))

print("The probability that the coin is fair (null hypothesis) is: ", p_value)
print("This means that the significance level (alpha) is: ", 1 - p_value)

The probability that the coin is fair (null hypothesis) is:  0.5101946756468801
This means that the significance level (alpha) is:  0.48980532435311985