Degrees of Freedom (Explained Simply)
A difficult concept with simple words.
Perhaps, one of the most “mysterious” concepts in statistics is “degrees of freedom”. When I studied hypothesis testing, I faced this notion several times but I couldn’t understand its meaning. Among books I’d read and web pages I’d browsed, no single one could give me a comprehension of what degrees of freedom do. Only after thorough research of tons of materials, did I get it.
In this article, I summarized my experience of deep-dive into degrees of freedom and I tried to explain this concept in a simple way, though without oversimplifying. I hope after reading there will be no reason to go somewhere else, unless you want to get a stricter explanation (in this way, I linked some research papers at the end of the article)
Let’s start with the definition.
Degrees of freedom - number of independent values (pieces of information), which were included into calculation of an estimate.
Wait… What does it all mean?
Basically, there are two terms: independent values and an estimate. An estimate is a single number that expresses some property of a population from a sample. It can be mean, median, standard variation, or variance. To get the estimate, we should calculate it by some formula. And there are independent values (or observations) that went into formula calculation. The quantity of these values is called “degrees of freedom”.
But what do we mean by saying “independent values”? It is better to explain this with examples.
Example 1. Suppose, your friend tosses the coin, and you should guess the outcome: tail or head. Of course, you can’t see the coin for some reason…let’s say, you’re turned back from it.
So, if after tossing your friend tells that the result was definitely not head, then you must conclude that the outcome was tail and nothing else. And vice versa. Therefore, we have only one independent observation — it is possible to choose one of two sides of the coin (head or tail), but once the side is chosen (for instance, tail), the other one must equal only one value (head).
Example 2. This time you and your friend decided to play another weird game called “guess the color of the traffic light”.
A traffic light has 3 colors:
⦁ red
⦁ yellow
⦁ green
You don’t see what the color is, and your friend tells you that the color is not yellow. So, you can choose between red and green. But the answer is still not clear. That’s why your friend gives a hint and says: “The color is not red”. Now, it is easy to “guess” that the traffic light can be only green. In this example, you were free to choose 2 outcomes out of 3, thus there are 2 independent pieces of information.
Example 3. And now, let’s move to the last example, which is very close to statistics. Suppose, we have some observations and know their mean.
We don’t know what value observation 2 has, but knowing the mean = 6 it is not hard to guess that observation 2 must be equal to 3 and not anything else.
So, this is the meaning of “independent” values. These values, or pieces of information that tell us about data, can be whatever they are, i. e. have the freedom to vary, while “dependent” values, as the name suggests, are dependent on free values. And degrees of freedom is the number that tells how many independent pieces of information went into the calculation of statistics.
Now, let’s answer the question: “Why degrees of freedom are important, and how are used?”.
One of the examples is the standard deviation of a sample.
This statistic tells how far the data is located from the sample mean on average. Under square root, there is a sum of squared differences from the sample mean x, divided by degrees of freedom: sample size — 1. It means that we have n — 1 independent pieces of information. You may ask: “Why do we subtract 1, but not 2, or another number?”. Well, in general, degrees of freedom equals the number of observations minus the number of parameters estimated. Because we’ve already calculated the mean, we can “guess” only one observation based on the rest data and the mean (see Example 3).
However, you may ask another question: “Why do we divide by degrees of freedom and not by the number of observations?”
This question is much harder to answer and that’s where I struggled the most. After diving into materials I began to see some common sense in this operation. I’ll be glad if after reading the explanation below you will see it too.
Suppose, we took a sample of 7 women from the population of 18–70 years old women living in San Francisco. Then, we measured their height. We got the following numbers of height in cm:
164, 173, 158, 179, 168, 187, 167.
Mean: 170.85
Now, we want to know how their height differs from the mean. We should calculate the standard deviation. But let’s calculate it using two slightly different formulas: in formula (1) we’re dividing by degrees of freedom, while in formula (2) we’re dividing by sample size.
The first formula gives us the next result:
SD ≈ 9.72
While the second one gives:
SD ≈ 9
It is easy to notice that when we divide by degrees of freedom, we make our estimate of standard deviation greater than if we were diving only by sample size. But why do we need to make it greater? As we’ve already calculated the mean, we don’t have to use all the data in order to calculate the standard deviation. It does not depend on each piece of information, and the last observation does not contribute to the standard deviation. So, if we don’t delete this redundant data, then we underestimate the standard deviation. Notice that degrees of freedom go to the calculation of standard deviation if it is estimated for sample data. The standard deviation of the population does not require degrees of freedom because the number of observations N is so big that N — 1 does not make any difference. Moreover, we can’t underestimate the actual standard deviation, because it is the actual standard deviation. In contrast, for small datasets, we have to include degrees of freedom in the calculation, as thus we can avoid underestimation of a parameter.
Thinking in this way can help you understand the importance of degrees of freedom in other statistical contexts, such as t-tests, chi-square tests, or regression. As I said, this is a difficult concept and even persons, who do statistics on a daily basis, are not always aware of what it means. So, unless you’re an aspiring Ph. D. in statistics, you can successfully analyze data without knowing what degrees of freedom are.
Yet, I hope this article helped you understand degrees of freedom. If after reading you have even more questions than before, please let me know in the comments.
REFERENCES
- R. H. Riffenburgh, Statistics in Medicine (Third Edition), 2012;
- Pandy, S., Bright, C. L., Social Work Research Vol 32, #2, June 2008.
- Walker, H. W. Degrees of Freedom. Journal of Educational Psychology. 31(4) (1940) 253-269
