What makes a matrix non-invertible?

Using Span & Linear Combinations to Understand Matrix Non-Invertibility

This article requires prerequisite understanding in basic linear algebra, and concepts such as linear combinations, span, matrix multiplication, and dot products.

I am currently sitting on the floor of my room. A few hours ago, I finished the third lecture in MIT OpenCourseware’s class 18.06.

This class, taught by professor Gilbert Strang, and recorded in the spring of 2005, has been one of my main learning tools for the past few days on my quest to better understand linear algebra for the purpose of machine learning. I’m accompanying the lectures by following along in Strang’s textbook Linear Algebra and it’s Applications (4th edition) and doing the exercises.

I’d like to offer some quick insight I was able to glean, as well as a few intuitions that I was able to come up with independently, on matrix non-invertibility.

This understanding requires a few prerequisites that I will only briefly review, such as the concept of span, and viewing systems of linear equations as matrices, and linear combinations, which are all pretty basic stuff.

Seeing matrix multiplication as a linear combination

Let’s modify our current understanding of a matrix multiplication. That is, taking a row in some matrix A and multiplying it by a column in some matrix B to get an item in AB.

Let’s abandon this typical view for now.

Let’s instead see how we can view this matrix multiplication in a slightly different way to get one column of C at a time.

Let’s look at a multiplication between two slightly larger matrices to make this new way a little more clear. This idea can be used in any matrices with agreeing dimensions (m x n) x (n x p), but we’ll stick to square matrices for simplicity.

How can we get C in one calculation?

Let’s see what happens when we use our old, familiar technique to multiply matrices to get this row. We would get this row by multiplying all rows of A by the first column of B.

Now, we have this computation matrix. But carefully inspecting each term in this matrix, can we express this in another way?

Notice that each “column” (column in quotes since it’s technically a vector) in the matrix on the right has a b term in common. From left to right, those are b11, b21, and b31. We can represent this in a more abbreviated form:

This, you should recognize as a linear combination, a combination of three vectors multiplied each by some scalar. These, like all linear combinations, have a span, which is, with any values of b, the amount of space the three vectors can describe. If all three of these vectors are independent, the span of these three vectors should be the entirety of 3-dimensional space.

Understanding non-invertibility

An inverse of a matrix satisfies the following equation, where the inverse of some matrix A multiplied by A is equal to the identity.

If you’re familiar with thinking of matrices as linear transformations, you can think of this as shifting space in accords to A and then switching back again to the default I using the inverse of A. If this sounds like nonsense, don’t worry about it.

Now, we can see when this doesn’t work. Let’s take a non-invertible matrix A.

But don’t take my word for it. How can we understand why this is non-invertible using linear combinations and span?

Let’s insert this A into the formula for invertibility. Pretend I haven’t already told you that it was non invertible.

We’re looking for some E, which when multiplied by A, returns the identity. This E would be the inverse of A. We know that there does not exist an E — but why?

Following the linear combination way of viewing matrix multiplication, we can view this multiplication as a linear combination.

Remember, the first column of I is a combination of the columns of A. Or more generally, the jth column of AB is a combination of the columns of A with each element in the jth column of B as scalars.

Let’s describe the first column of the identity I using this column form of multiplication.

Similarly, we can get the second column of the identity by using the second column of E instead.

Since the vectors stay the same, and thus describe two identical combinations which share the same span, we can compound it into the following statement.

After interpreting the matrix multiplication as a linear combination, we now show graphically, that E, our non-invertible matrix, does not contain either (1, 0) or (0, 1) in it’s span.

The yellow vector is (3, 6), the red (1, 2), the green and blue the columns of the identity matrix (0, 1) , (1, 0). The dashed line represents the span of these two vectors.

Since the two vectors are colinear (parallel), the span of their combinations is 1 dimensional — only on the grey dotted line. No scalars multiplied by those vectors can create a new vector that points anywhere else — including the two identity vectors, which are not in the span of our combination.

This same logic could be applied to three dimensional non-invertible matrix. In this scenario, the columns of our 3 x 3 identity matrix I, namely (1, 0, 0), (0, 1, 0) and (0, 0, 1), would not all be covered by the span. If one column in our non-invertible matrix is dependent, our span would only be a 2D plane in 3D space. It will be impossible for all these three basis vectors (the columns of the identity matrix) will not be in that span.

If the span of a system with one dependent vector is represented with the red plane, then it is impossible that all three basis vectors can be included within that span. The maximum would be two, if the plane was lying flat along one of the axes.

With a deeper geometric understanding of non-invertibility, concepts that go into finding the inverses of invertible matrices, like Gauss-Jordan elimination, will become a lot easier to understand.

Adam Dhalla is a high school student out of Vancouver, British Columbia, currently in the STEM and business fellowship TKS. He is fascinated with the outdoor world, and is currently learning about emerging technologies for an environmental purpose. To keep up,

Follow his Instagram, and his LinkedIn. For more, similar content, subscribe to his newsletter here.