Linear Algebra 101 — Part 1

I believe understanding fundamental concepts is crucial when it comes to learning something advanced. Why? Because the fundamentals are the basis where you build your advanced knowledge on top of. If you put more things on top of the weak basis, it could break apart in the end, meaning you end up not fully understanding any of the materials you learned. Then you might need to go back again to learn the fundamentals before going back to learn the most exciting advanced materials which could be time consuming.

Linear Algebra is one of the fundamental topics that you should be very comfortable with. I’m particularly interested in Brain Computer Interface (BCI) and many fields composing the field use Linear Algebra as a basis (e.g. Digital Signal Processing, State-space estimation, Machine Learning, Deep Learning, etc.).

In this series “Towards understanding Linear Algebra”, I’m going to cover topics based on famous and excellent lectures by Dr. Gilbert Strang. It’s a very good lecture and I strongly advise you to watch all the videos. My purpose here is to provide you with enough details to understand the key concepts from his lectures so that you could gain the most out of it with the smallest amount of time.

Enough said, let’s get started! 👍

Materials covered in Part 1:

I’m starting off the series of articles with the following topics in Linear Algebra:

• Matrix elimination (Gaussian elimination)
• Row echelon form
• Reduced row echelon form (rref)
• Vector space
• Subspace

If you know all the above contents, you are good to go. I have a summary of these at the bottom for you to recap these key concepts quickly.

Introduction to row elimination and reduced row echelon form

Do you remember when you were in elementary school or junior high school trying to solve something like below in your math class?

To solve this, I bet you were doing something like this:

That was easy enough. Let’s try solving this using matrices. This is the first step in Linear Algebra. Doing eliminations.

First, we need to express the original equations in a matrix form.

You can solve this in a very similar manner as you did in the beginning. Using Gaussian elimination.

Gaussian elimination - Wikipedia

This special form of combining the matrix you want to solve and the output with “|” and performing row reduction is often referred to as “Gaussian Elimination”.

It’s very similar to what we did in the quadratic equation but you see we left “1” at the top row and made the 2nd row 1st element “0”? This is called “row echelon form (ref)” . By its definition, row echelon form satisfies the following conditions (from Wikipedia):

• All nonzero rows (rows with at least one nonzero element) are above any rows of all zeroes (all zero rows, if any, belong at the bottom of the matrix)
• The pivot (the first nonzero number from the left) of a nonzero row is always strictly to the right of the pivot of the row above it (some texts add the condition that the leading coefficient must be 1).

Row echelon form - Wikipedia

Then from here, we can solve the equations as follows:

You could solve like this, but people usually won’t solve this way.

Apparently, this is not a good way to solve in Linear Algebra because you are still doing some calculations once you got back to quadratic form. Isn’t there a way to completely solve this just by row elimination? There is!

Note that you could divide a particular row like this and try to make “pivots” = 1 in all the rows. This form in the left side of | is something called “reduced row echelon form (rref)” and bringing the combined matrix (your initial matrix and the output) down to this form is called “Gauss-Jordan elimination”.

Let’s summarize the reduced row echelon form (rref) by definition because it’s one of the most important form here:

• It is in row echelon form.
• Every pivot is 1 and is the only nonzero entry in its column.

Row echelon form - Wikipedia

Introduction to vector space

So now we know that we could solve the familiar quadratic equations in a Linear Algebra way, but have you ever thought about what the answers indicate in geometry? (x=2 and y=1)

Let’s change the original quadratic equation into different form.

This form shows that you have 2 columns each multiplied by “x” and “y” to get a column vector (11, 6). Note that you could decompose any matrix multiplication like this = summation of column vectors with some coefficients.

Let’s see how these column vectors appears in a graph.

Since we have x = 2 and y = 1 as a solution, we are multiplying the 1st column vector by 2 (lengthening by 2).

And summation of vectors means move one resulting vector in parallel and connect with the other vector to draw a line from the origin. This gives us the output of (11, 6). If you think about it, 2 * (3, 1) = (6, 2) and adding that to (5, 4) gives us (11, 6).

Now you have actually stepped into something called “vector space”! Vector space is a space represented by vectors (in our case, column vectors). It’s not limited to the 2D example I just showed you above.

Vector space - Wikipedia

Like in the example above, the vector space must satisfy vector addition (like when we added two vectors to get the outcome) and multiplication by a scalar (like when we multiplied the 1st column vector with a scalar 2) and stay in the same vector space.

• The first operation, called vector addition or simply addition + : V × V → V, takes any two vectors v and wand assigns to them a third vector which is commonly written as v + w, and called the sum of these two vectors. (Note that the resultant vector is also an element of the set V ).
• The second operation, called scalar multiplication · : F × V → V， takes any scalar a and any vector v and gives another vector av. (Similarly, the vector av is an element of the set V ).

Introduction to subspace

And there’s another important concept called “subspace”, sometimes referred to as “linear subspace”.

Subspace is a little tricky one. It’s a vector space inside a vector space. Did I confuse you? Sorry about that. Let me explain in more detail.

In the above example I showed you, we have 2 column vectors. These column vectors could actually represent any points on the 2D plain. Let’s say you want (0, 0) then we just simply set x = 0 and y =0. Let’s take a little more complex example, say you want (1, 2). No problem. If you solve like what we just did, we get x = -6/7 and y = 5/7. So these 2 column vectors are making this vector space. Think of it like a plane, a sheet of paper covering up the axis. No matter where you draw a point on the paper, there is a way to represent that with the 2 column vectors.

Subspace is a space in a vector space. If you think the above example as a subspace, then the subspace is inside some other (bigger or larger) vector space. Go back to the sheet of paper example. If you have a sheet of paper in front of you, touch it, lift it up, rotate. Yes, the sheet of paper is 2D space but the place where you are playing with that piece of paper is in a 3D world. In this case, the 2D space on a paper is subspace and the world you are living in is a larger vector space.

I used the above example to just give you an idea, but there’s certain rules that subspace must meet.

Here’s the definition: Let K be a field (such as the real numbers), V be a vector space over K, and let W be a subset of V. Then W is a subspace if:

1. The zero vector, 0, is in W.
2. If u and v are elements of W, then the sum u + v is an element of W.
3. If u is an element of W and c is a scalar from K, then the scalar product cu is an element of W.

If you go back to the original example I gave you above, think of W as the sheet of paper where I drew the axis with 2 column vectors. It has to have zero vector (origin) as stated in the 1st definition. You could think u and v as the 2 column vectors that we discussed. Does it become clear? I bet not at the first attempt 😔

Linear subspace - Wikipedia

Don’t worry. Sometimes grasping a certain concept takes time. My advice is to not only just focus on one material and trying to understand, but instead, go through as much material on the same topic that you are trying to understand so that you get different points of views on the same topic. This will help you understand the concept. I’m posting a video lecture here from MIT open courseware but you could also try googling “linear subspace” to access tons of materials. I suggest you spend some time on this topic before going further.

Alright. Now, that was a vector space and a subspace. In the next article, I’m going to talk about nullspace and rank.

Summary

• Matrix elimination (Gaussian elimination)

It’s a way to perform reduction per row to transform the matrix into the form that’s easier for you to utilize.

• Row echelon form

Row echelon form must satisfy the two conditions: 1) All nonzero rows are above any rows of all zeroes. 2) The pivot of a nonzero row is always strictly to the right of the pivot of the row above it.

• Reduced row echelon form (rref)

Reduced row echelon form (rref) must satisfy the two conditions: 1) It is in row echelon form. 2) Every pivot is 1 and is the only nonzero entry in its column.

• Vector space

It’s a space represented by a linear combinations of vectors.

• Subspace

The definition of a subspace is a subset that itself is a vector space.

I hope this helps and stay tuned for more exciting topics about Linear Algebra!

Towards understanding Linear Algebra — Part 2