Exploring the Importance of Hypothesis Testing in Data Science & Finance through simple examples in Python
Introduction
In this article, I will discuss and provide examples in python of different types of hypothesis tests and highlight their significance in the fields of data science and artificial intelligence.
Hypothesis testing is an essential tool in data science as it provides a systematic and objective way to test claims and make decisions based on data. It allows data scientists to determine the validity of their assumptions and beliefs about the relationships between variables in a dataset. This helps them to make informed critical decisions, draw meaningful conclusions, and identify patterns in the data that may not have been immediately obvious. In addition, hypothesis testing is also used in hypothesis-driven machine learning and modeling, where it helps to evaluate the performance of models and make decisions about the selection and refinement of models. Therefore, hypothesis testing is a critical component of data science that helps to ensure the accuracy, reliability, and robustness of data-driven decision making.
What is hypothesis testing in Mathematics
Hypothesis testing is a technique in statistics for evaluating the validity of a statement about a population, based on sample information. The process entails formulating two conflicting hypotheses regarding the population parameter, then using the sample data to determine the hypothesis that is most probable. This hypothesis is regarded as the “null hypothesis,” with the other referred to as the “alternative hypothesis.” The results of the hypothesis test are then utilized to reach a conclusion regarding the population parameter and either support or reject the null hypothesis.
Here are some common hypothesis tests used in the beautiful field of Data Science :
1. Z-test
I think everyone knows about this test, it shows up almost every time in any statistical foundation class .
Z-test is simply used to test the mean of a single sample to determine if it is significantly different from a known population mean .
When to use the Z-test :
The test is applicable when you have a single sample and the standard deviation of the population is known.
Python implementation
Here is a straightforward Python code that implements a z-test hypothesis test :
from scipy import stats
import scipy
import numpy as np
# sample and known population mean & standard_deviation
sample = [22,11,9,12]
population_mean = 20
population_standard_deviation = 15
#calculate sample mean and standard deviation
sample_mean = (sum(sample)/len(sample)) # Mean , you can also use np.mean() if you choose to do so
sample_standard_deviation = np.std(sample)
#compute Z-statistic
z_statistic = (sample_mean - population_mean) / (population_standard_deviation / np.sqrt(len(sample)))
#compute P-Value
p_value = stats.norm.sf(abs(z_statistic))
#interpret results
if p_value < 0.05 :
print(f'Reject null hypothesis,sample mean is significantly different from population mean , P-value:{p_value}')
else:
print(f'Fail to Reject null hypothesis,sample mean is not significantly different from population mean, P-value:{p_value}')
Output:
Fail to Reject null hypothesis,sample mean is not significantly different from population mean, P-value:0.1930623371419069
2. ANOVA
ANOVA, short for Analysis of Variance, is a statistical technique used to compare the means of more than two groups or samples to determine if there is a significant difference among them.
When to use the ANOVA test :
This test is adopted when you have more than two independent samples or groups and you want to compare their means to determine if there is a significant difference between any of the groups.
Python implementation
Here is a straightforward Python code that implements a ANOVA hypothesis test :
import pandas as pd
import statsmodels.api as sm
import statsmodels.formula.api as ols
#defining three independent groups
group1 = [2,9,7,6,5]
group2 = [2,3,1,3,6]
group3 = [1,2,3,2,1]
# Let's create a DataFrame in pandas
data = {'group':[1,1,1,1,1,2,2,2,2,2,3,3,3,3,3],'value': group1 +group2 + group3 }
df = pd.DataFrame(data)
#Let's compute ANOVA
ols_model = sm.formula.ols('value ~ C(group)', df).fit()
anova_table = sm.stats.anova_lm(ols_model, typ=2)
#interpret results
if anova_table['PR(>F)'][0] < 0.05:
print(f'Reject null hypothesis,There is significant different between the group mean,P-value: {anova_table["PR(>F)"][0]}')
else:
print(f'Fail to Reject null hypothesis,There is no significant different from population mean,P-value: {anova_table["PR(>F)"][0]}')
Output:
Reject null hypothesis,There is significant different between the group mean,P-value: 0.01729898724329134
3. Chi-Squared test
This one is an interesting test , assume you have categorical variables that you want to test the association between them , then the Chi-squared test is the most suited hypothesis test in this case.
Python implementation
Here is a straightforward Python code that implements a Chi-squared hypothesis test:
from scipy import stats
import numpy as np
#Contingency table
table = np.array([[30,40], [20,60]])
#running the Chi test
chi2_stats, p_value, _, _, = stats.chi2_contingency(table)
#interpret results
if p_value < 0.05:
print(f'Reject null hypothesis,There is significant association between the two categorial variables,P-value: {p_value}')
else:
print(f'Fail to Reject null hypothesis,There is no significant association between the two categorial variables,P-value: {p_value}')
Output:
Reject null hypothesis,There is significant association between the two categorial variables,P-value: 0.03227734609479707
4. Student’s t-test
This hypothesis test is used to compare the means of two independent samples to determine if there is a significant difference between them.
When to use the t-test :
You use it when you have two independent samples and the population standard deviation is unknown . This hypothesis is usually adopted when the sample size is small (n < 30).
Python implementation
Here is a straightforward Python code that implements a t-test hypothesis test:
# Student's t-test
from scipy import stats
#define two independent samples
sample_A = [15,31,22,24,28]
sample_B = [5,11,20,23,27]
# performing the t-test
t_stats, p_value = stats.ttest_ind(sample_A,sample_B)
#Let's create a condition
if p_value < 0.05 :
print(f'Reject null hypothesis,sample is significantly different, P-value:{p_value}')
else:
print(f'Fail to Reject null hypothesis,sample is not significantly different, P-value:{p_value}')
Output:
Fail to Reject null hypothesis,sample is not significantly different, P-value:0.20034521767305966
5. Paired t-test
This test is used to compare the means of two dependent or related samples to determine if there is a significant different between them .
When to use the Paired t-test :
The paired t-test is used to compare the means of two related or dependent groups of samples. This test is appropriate to use when the samples are paired or matched in some way (e.g. before-and-after measurements on the same individuals, or measurements on matched pairs of individuals, etc.). The paired t-test assumes that the differences between the paired observations are normally distributed.
Python implementation
Here is a straightforward Python code that implements a paired hypothesis test:
from scipy import stats
import numpy as np
# Paired samples
inital = [2,5,8,9,6,]
final = [6,7,8,9,3]
#perform paired t-test
t_stats, p_value = stats.ttest_rel(inital,final)
#interpret results
if p_value < 0.05 :
print(f'Reject null hypothesis,the changes are significantly different, P-value:{p_value}')
else:
print(f'Fail to Reject null hypothesis,the changes are not significantly different, P-value:{p_value}')
Output:
Fail to Reject null hypothesis,the changes are not significantly different, P-value:0.6340271611962769
6. McNemar’s test
This test compared two dependent proportions with each other or dependent groups.
When to use the Paired McNemar’s test :
McNemar’s test is appropriate to use when the two groups are dependent or matched (e.g., the same individuals are tested twice under different conditions). McNemar’s test is often used in medical studies to compare the efficacy of two treatments or to test for a change in a binary variable over time.
Python implementation
Here is a straightforward Python code that implements a McNemar’s hypothesis test:
import numpy as np
from statsmodels.stats.contingency_tables import mcnemar
table = np.array([[1, 0, 1, 1, 0], [1, 1, 0, 0, 1]])
#running the McNemar's test
mnemar_stats, p_value, _, _ = mcnemar(table)
#interpret results
if p_value < 0.05:
print(f'Reject null hypothesis,There is significant association between the two categorical variables,P-value: {p_value}')
else:
print(f'Fail to Reject null hypothesis,There is no significant association between the two categorical variables,P-value: {p_value}')
7. Wilcoxon rank-sum test
The Wilcoxon rank-sum test is a non-parametric statistical test used to compare two independent samples. It is used to determine if there is a significant difference between the medians of two groups. Unlike the t-test, which assumes normality of the data, the Wilcoxon rank-sum test does not make any assumptions about the underlying distribution of the data, making it a useful test for comparing samples from non-normal populations.
When to use the Wilcoxon rank-sum test :
Wilcoxon rank-sum test can be used in situations where the assumptions of normality are not met or when the sample sizes are small. It is often used in biomedical and social sciences research to compare the distributions of two independent groups of continuous or ordinal data.
Python implementation
Here is a straightforward Python code that implements a McNemar’s hypothesis test:
from scipy.stats import mannwhitneyu
def wilcoxon_rank_sum_test(data1, data2):
stat, p_value = mannwhitneyu(data1, data2)
return p_value
data1 = [1, 2, 3, 4, 5]
data2 = [6, 7, 8, 9, 10]
p_value = wilcoxon_rank_sum_test(data1, data2)
if p_value < 0.05 :
print(f'Reject null hypothesis,There is significant different in medians between the two groups, P-value:{p_value}')
else:
print(f'Fail to Reject null hypothesis,There is no significant different in medians between the two groups, P-value:{p_value}')
Output:
Reject null hypothesis,There is significant different in medians between the two groups, P-value:0.007936507936507936
Here are some examples of hypothesis testing in finance applications:
- Testing the efficiency of financial markets: This involves testing the hypothesis that financial markets are efficient, meaning that prices accurately reflect all available information.
- Testing the validity of financial models: This involves testing the hypothesis that a particular financial model accurately represents the underlying financial data.
- Testing the reliability of financial forecasts: This involves testing the hypothesis that a financial forecast is accurate and reliable.
- Testing the validity of trading strategies: This involves testing the hypothesis that a particular trading strategy is profitable and effective.
- Testing the relationship between financial variables: This involves testing the hypothesis that two or more financial variables are related, such as the relationship between stock prices and interest rates.
- Testing the validity of credit scoring models: This involves testing the hypothesis that a credit scoring model accurately predicts the creditworthiness of borrowers.
- Testing the performance of investment portfolios: This involves testing the hypothesis that a particular investment portfolio outperforms a benchmark portfolio.
These are just a few examples of how hypothesis testing is used in finance applications. The specific hypotheses and methods used will depend on the particular financial problem being studied.
Thanks for taking the time to read this article and I hope you enjoyed it , please don’t hesitate to connect with me for any questions regarding this article.Also don’t forget to follow me for more Data Science related articles.