Understand the Test Statistics: from A Likelihood Ratio Perspective

TL;DR

I always feel excited when two methods come from different perspectives, but coincide to the same conclusion in the end. In this article, I want to show how some most commonly used test statistics can be derived from a likelihood ratio perspective. It not only helps improve my understanding on the hypothesis testing, but also strengthen my statistical thinking.

Background

Hypothesis testing is one of the most commonly used techniques in various industries, to evaluate performances of new product designs, marketing campaigns, ML models, etc.

Some of the popular tests (e.g., z-test and t-test) provide nice analytical formulas for their test statistics, and we can easily plug in observed data into these formulas to make test conclusions (i.e., reject or fail to reject the null hypothesis).

What is Test Statistic

This wiki page gives a comprehensive explanation on test statistic. In simple and plain words, a test statistic

1. is a single value that summarizes the sample data, and
2. quantifies how likely to observe the sample data if the null hypothesis is true.

The test statistic distinguishes the null hypothesis from the alternative hypothesis.

What is Likelihood Ratio Test

Likelihood ratio test is a test procedure for hypothesis testing. It uses the ratio (i.e., the likelihood ratio) between the two likelihood below as its test statistic.

1. Maximum likelihood of the sample data given the null hypothesis is true
2. Maximum likelihood of the sample dataNote that the second likelihood is independent of the definition of the null and alternative hypotheses, and only depends on the distribution and the sample data.

The lower this likelihood ratio is, the more extreme the observed sample data is, and accordingly, the less likely our null hypothesis is true. If we have a predefined significant level, then it corresponds to a threshold of this likelihood ratio, below which we should reject the null hypothesis.

The likelihood ratio test procedure is outlined below:

Examples

Let’s use some popular tests to show that their test statistics can be derived from the likelihood ratio procedure described above.

Z-test

T-test

Some Additional Thoughts

Deriving these test statistics using the likelihood ratio procedure shows that hypothesis testing follows one of the typical decision making processes:

1. Make some reasonable assumptions (e.g., normal distribution, independent samples).
2. Formulate and solve an optimization problem (e.g., maximize the sample likelihood with and without the condition of H0 is true).
3. Make a decision based on the optimal solutions of the problem (e.g., compare likelihood ratio and a predefined threshold).

Remarks: I tried to reduce the math as much as possible while keeping the backbone of the deductions, but it still ends up with a bunch of formulas. Also, due to the lack of math support in Medium, I have to put some screenshots from my LaTeX document here.