How To Understand Linear Independence (Linear Algebra)
In a linear algebra college course, you get bombarded by definitions. A mini-dictionary of terms with confusing, arcane definitions riddle the textbook, and your typical college student only has a couple of months to understand eigenvalues, eigenvectors, and Hermitian matrices (and of course, more) before their tests.
It’s also discouraging because we are usually not taught intuition in these classes. Instead, we stare at matrices, eventually learning through trial and error how to get the right answers without understanding why we use the methods we use. Linear algebra is actually a deeply interesting branch of math, but when we mindlessly gaze for hours at matrices, we don’t get an understanding — we get eye strain.
So, I’m going to give you an analogy that I used in my tutoring sessions to explain some linear algebra concepts. Namely, these concepts:
- Linear dependence/linear independence
- Span
- Basis
There will be 3 posts, one post for each concept. This post explains linear dependence/independence intuitively, using the analogy of painting. Span and basis will be in future posts. Note, I’m mostly going over the concept of dependence/independence and not the process of how to solve linear algebra questions.
I am going to assume that you’re familiar with how vectors are written (that is, as a column of vertical numbers), adding vectors together, and the idea of linear combinations. Let’s get started.
The Paint Analogy: Linear Dependence
Suppose you’re a painter, but not the kind that cuts your own ear off. If I gave you three paint buckets, with red, blue, and purple paint, is it possible to mix two of those colors together to get the third color?
Clearly the answer is yes: combining the red and blue paint gives you purple. When there is a way to combine amounts of two colors to create a third color, we might say that the three colors together are dependent on each other.
If I’m stingy and trying to save money on paint, I could get away with only buying red and blue paint instead of red, blue, and purple, because I can just mix red and blue paint to get purple paint. So, you might say that a consequence of dependent colors is that at least one color is unnecessary (in this case purple).
We could also use different amounts of each color, not just a 1:1 ratio. If I gave you three paint buckets with red, white, and a very light hue of pink, you might need 1 part red paint and 16 parts white paint to get that specific shade of pink. Those colors are still dependent on each other, even if the combination of red and white paint is not a 50–50 split.
Similar to the paint buckets, linear dependence in vectors comes from combining vectors to get other vectors. Say that I give you a few 2-dimensional vectors (which you can think of as coordinates in 2-D space) called v₁, v₂, and w. These vectors are drawn in the diagram below. Question: is there a way to combine multiples of v₁ and v₂ (orange vectors) to create w (green vector)?
Yes. In this case, if you multiply both v₁ and v₂ by 2, then add them together, it equals the vector w. Kind of like mixing red and blue to get purple.
Here, w is what we call a linear combination of v₁ and v₂. Since it is possible to combine two of these vectors to get the third vector, we would call this sequence of vectors linearly dependent.
Linear Independence
Going back to the paint, let’s say that I gave you red, blue, and yellow paint. Are these colors dependent? Is there a way to mix two of these colors into the third color? In this case, no. No amount of red and yellow will create blue; you just get varying shades of orange. Similarly, no matter how much you mix red and blue together, you are never going to create yellow paint. If you keep trying you might even get so frustrated you find yourself with one ear. These three colors can’t be dependent on each other, so we could call them independent.
With vectors, independence (aka linear independence) is similar but a little trickier, so let’s start with this: is there a way to combine v₁ (0,1) and v₂ (1,0) to get some multiple of w (2,2)? Multiples of w would include (4,4) or (-2,-2) for example, and in essence this question asks if there are values of c₁, c₂, and c₃ that make the following equation work:
The answer is yes for pretty much any multiple of w. For example, if I wanted to combine v₁ and v₂ to get (4,4), I can take 4(v₁)+4(v₂) to get the solution. In this case c₁ equals 4, c₂ equals 4, and c3 equals 2. But if you’re sly, you might notice that (0,0) is technically a multiple of w as well. We could just multiply v₁, v₂, and w by 0 to get the following, true equation:
This “solution” of multiplying everything by zero is called the trivial solution. It’s trivial in the sense that it works no matter what vectors I give you. It doesn’t matter if the vectors are dependent or independent, the trivial solution will always work, so in this sense the trivial solution doesn’t help us.
However, the two other solutions we’ve used before, 4(v₁)+4(v₂)=2(w) and 2(v₁)+2(v₂)=w, are non-trivial solutions, which just means they’re not the trivial solution (all zeroes).
If I’ve got some sequence of vectors, like (v₁,v₂,v₃,v₄,v₅), and I want to combine some of those vectors to get a multiple of another vector, it’s the same as solving one of these two equations (both equations mean the same thing):
If the vectors have a non-trivial solution to one of these equations, those vectors are linearly dependent. However, if there isn’t a non-trivial solution, the sequence is linearly independent. The trivial solution is a solution regardless of independence.
The vectors from our earlier example, (v₁, v₂, and w) would be linearly dependent. On the other hand, (v₁,v₂) by themselves are linearly independent, because you can’t scale (0,1) to get to (1,0) without the trivial solution. .Trying to scale v₁ to get v₂ would be like mixing red paint to create blue.
Negative Paint
There is one part where the analogy falls apart a little, and that should be explained. With dependent colors of paint, red+blue makes purple, but purple+blue doesn’t make red. But in linearly dependent vectors, any of the vectors can be expressed as a combination of the others.
I already showed that w is a combination of v₁ and v₂, but v₁ is also a combination of v₂ and w; it’s just a little less obvious.
One way to get around this is to imagine a way to remove an amount of paint from a mixture (aka adding a negative amount). We can then express purple as 1(red) + 1(blue), and then blue as 1(purple) -1(red). This would mean taking the purple paint and removing one part red paint, leaving blue.
Short Review
- Given the equation c₁v₁ + c₂v₂ + … + cnvn = 0v, where all the v’s are vectors, 0v is the zero vector, and the c’s are scalars, then setting all the c’s to zero is called the trivial solution.
- A set of vectors is linearly dependent if there exists a non-trivial solution to the equation c₁v₁ + c₂v₂ + … + cnvn = 0v (which actually implies infinite solutions).
- A set of vectors are linearly independent if there doesn’t exist non-trivial solutions to the equation c₁v₁ + c₂v₂ + … + cnvn = 0v
Other Ways to Find Linear Dependence/Independence Not Discussed In General
- If you put a set of vectors into a matrix, the vectors are independent if and only if every column is a pivot column.
- Another way of saying the above point: if you put your set of matrix and solve the system of linear equations, dependent vectors will have infinite solutions, but independent vectors will only have one solution (that sneaky trivial solution)
- A set of vectors is dependent if there are more columns than rows when you put them into a matrix.
- Another way of saying the above point: if there are more vectors than the dimension of your space, they’re dependent. For example, if your vectors are in 2-D space, two vectors may or may not be dependent, but three vectors or four vectors will always be dependent.
Conclusion
A big reason linear dependence is important is because if two (or more) vectors are dependent, then one of them is unnecessary, since the span of the two vectors would be the same as the span of one of the two vectors on their own (and again, span will be covered in a different post). This is like removing purple from your palette because you already have red and blue.
While linear dependence and independence aren’t usually explained in this way, it’s helpful to get a deep understanding of this concepts. It widens the scope of your linear algebra understanding. It helps you get clarity. It’s like looking at a painting of the horizon, with crimson sunsets and an inky black river. You could marvel at the thousands and thousands of colors the painter had to use to make their painting. Or, you could notice that they were only using red yellow and blue the entire time.
