Hypothesis testing in real-life research as opposed to in textbooks on statistics
According to most textbooks on statistics, the hypothesis testing procedure is an overly formalistic procedure: the assumptions (random sample, type of variables, statistical technique, alpha level), the two hypotheses (the null and the alternative), the results of the statistical analysis, the calculation of a test statistic, the evaluation of the p-value, and the drawing of a conclusion and/or deciding. In real-life research (at least in the social and behavioral sciences widely defined), my experience is that this procedure in most cases is much more straightforward — if not to say trivial. Let me explain by a few examples, where I keep the technicalities at a minimum. Our variables concern some aspects of a random sample of 300 restaurant meals consumed in a pizza restaurant chain — such as the total bill for the meal and the number of diners present at the table. The average total (table) bill for the meals in the sample is 52.3 Euro, and these bills range from 10.5 to 229.8 Euro. Figure 1 presents the histogram for the total bill variable.
Hypothesis test 1: comparing two means
Let’s say we want to find out if the total bill for meals including alcohol amount to the same on average as meals not including alcohol. In cases such as these, the null hypothesis is always that meals with alcohol (group A) and without alcohol (group B) cost the same on average. Stata-exhibit 1 shows the necessary results to test this null hypothesis.
We note that meals with alcohol cost 57.85 Euro on average, and that meals without alcohol costs 49.72 Euro. The average difference in the sample is thus 8.12 Euro (or -8.12); see also the “diff” row in red. The test-statistic for this difference is 2.05 (or -2.05); see also the “t =” in red. If you don’t know what a t-value is, no worries, you do not need to! The reason is that a t-value always “translates” into a p-value, which we find at the bottom of the exhibit in red in the middle: 0.0404.
The p-value is one of the most misunderstood statistical concepts of all time, but let’s not get into that now. What it means in our example is this: if the meals with and without alcohol in the unknown population cost the same on average (i.e., what the null hypothesis implies), the probability of observing a 8.12 Euro difference between the two groups of meals, or a larger one than this, in a random sample from this population is 4.04% (0.0404 = 4.04%). That is, this probability is smaller than 5%, which is the typical convention in statistics to suggest that there is a statistically significant association between two variables in the population (i.e., the 5% significance level). In other words, we reject the null hypothesis and conclude that meals with alcohol cost more on average in the population than meals without alcohol. We thus get indirect support for our alternative hypothesis, namely that our two group means are unequal in some way. (Note! The p-value says absolutely nothing about the probability of the alternative hypothesis.)
Hypothesis test 2: regression analysis
Suppose we want to know how the number of diners at the table is associated with the total bill. Stata-exhibit 2 presents the regression analysis in question. (If you do not know anything about linear regression analysis, please see my post here or here.)
The exhibit shows that the regression coefficient is 12.5, suggesting that one more diner at the table implies a 12.5 Euro larger total bill. In our case the null hypothesis is this: one more diner at the table implies a zero Euro larger total bill (i.e., a regression coefficient of zero). We note the t-value in red (12.99) and its associated p-value in red (“P>|t|”): 0.000, meaning less than 0.0001. In other words: If the regression coefficient in the unknown population is zero (i.e., what the null hypothesis implies), the probability of observing a regression coefficient of 12.5 Euro, or a larger one, in a random sample from this population is less than 0.01%. This number is much smaller than 5%, and we thus reject the null hypothesis. We once more get indirect support for our alternative hypothesis, namely there is a (statistically significant) association in the population between the two variables under scrutiny. BTW, nothing really new happens for a hypothesis test in the multiple regression framework.
Hypothesis test 2: cross-table
We wonder if meals with or without alcohol have the same probability of inducing a tip. Stata-exhibit 3 presents the cross-tabulation. (If you are rusty on cross-tabulations, please see my post here.)
On the right-hand side of the output (“Total”), we see that 45% of all the meals included a tip. The null hypothesis is that meals with or without alcohol have the same tipping probability. Yet the sample probabilities are 52% (with alcohol) and 42% (without alcohol) — a 10-percentage points difference. In this example, the p-value (now the result of a chi-square test and not a t-test) expresses the probability of finding this 10-percentage points difference, or a larger one, if the percentage points difference in the unknown population is zero. This p-value is 11%, as shown at the bottom of the output in red. We thus cannot reject the null hypothesis suggesting equal tip probabilities, and we must keep it: There is no (statistically significant) association between the two variables in the population.
Takeaways and implications
The upshot of this post is straightforward to summarize: In practical research in the social and behavioral sciences widely defined, the way to judge a statistical association as “statistically significant” or not is typically much less strenuous than what it appears like in textbooks on the subject. This does not mean that hypothesis testing in general is easy (and far from it!), but that’s a topic for another blog post.
About me
I’m Christer Thrane, a sociologist and professor at Inland University College, Norway. I have written two textbooks on applied regression modeling and applied statistical modeling. Both are published by Routledge, and you find them here and here. I am on ResearchGate here, and you also reach me at christer.thrane@inn.no