Entering the Matrix: ELI5 Introduction to Eigenvalues and Eigenvectors
This post is motivated by my effort to explain matrices and their special properties without having to resort to typical mathematical language.
Matrices are a strange beast, look mathematical devoid of all physical intuition, and yet are so powerful that they underlie foundational mathematics, statistics and even quantum mechanics. Lets bring some intuition to their working.
But first, free your mind of any associations that you may have with mathematical terminologies of vectors, spaces, basis or even matrices. Lets delve into a magical world with our protagonist, Wincerind, an acquaintance at best of another colorful fella.
Lets imagine a magical compass.
Wincerind finds it in an old witch’s home, but doesn’t know what that compass does, or what its powers are. In turmoil, he turn to sleep hoping for an answer the next day, like all great men often do. In his dreams, Morpheus appears and reveals to him that the compass is a size changer.
“What?”, Wincerind awakes, hearing the magical cauldron he put on heat last night burst. He wonders, “a compass that is a size changer, hmm.”
Tired by his forced late night cauldron cleaning, Wincerind falls into another deep slumber. The Oracle appears in his dreams and says, “Morpheus is a fool, he doesn’t know the true nature of this compass. This compass is a direction changer.”
“Not again!”, Wincerind shouts, hearing the whistle of the kettle that he put on.
He is too awake now to do anything but test who is correct, Morpheus or Oracle.
In a true scientific manner, Wincerind decides to test the effect of the compass. He decides to use the magical compass on a VicVac. He looks into his basket of magical creatures and find a small VicVac. For those unfamiliar, VicVacs are magical beasts that are composed of parts of different animals. The one in his basket is one part ibex and two parts jackal.
Step 1: Preparation
Wincerind is happy to find such a small VicVac is his basket of wonders. He is too tired to handle a big one. He makes a note in his enchanted diary:
“VicVac in hand = (1i, 2j); where i denotes number of ibex parts and j denotes number of jackal parts.”
Step 2: Setup
Wincerind brings the magical compass [MC] close to this VicVac. He scribbles the following note in his diary:
Step 3: Observation
Lo and behold! Wincerind observes that his simple VicVac has grown fatter and taller. He struggles to control it, and has to resort to his potion of drowsiness. Upon inspection, Wincerind find that the new VicVac now has three parts ibex and four parts jackal. He makes the following entry in his enchanted diary:
Wincerind thinks on what happened with the use of the magical compass. He is sure of two things at this point,
- The magical compass is a ‘size changing’ device. It increased the size of the Vic Vac. (Which also means Morpheus in Wincerind’s dreams was correct)
- The magical compass increased the ibex parts 3 times (i.e. from one to three) while the jackal parts only 2 times (i.e. from two to four).
At this point, any layman would conclude Morpheus was correct, the compass is a size changer. But Wincerind is too good a scientist to jump to conclusions. He decides to graphically look at his results to understand the compass behavior.
Step 4: Analysis
To analyse his findings, he decides to plot #parts of ibex horizontally and #parts jackal vertically.
He finds that the original VicVac (V, in blue) grew in size to new VicVac (V’, in red) and the effect of Magic Compass was to change size of V .
But,
Not only did V’ become larger in size, it also was changed its direction at an angle from the original V.
Wincerind is amazed! The Oracle was right, Morpheus didnt know the true nature of the compass.
Step 5: Communicating Knowledge
Wincerind decides to explain to Morpheus why he was wrong. Tired by his hectic morning activities, he doesnt have to wait long before Morpheus appears in his dreams.
“Yo Morpheus, let me explain to you the true nature of the compass. Lets break down the graphical behavior of magical compass into two steps.
A. Pure size change
Lets design a first step, in which the magical compass multiplies both the ibex and jackal parts by the same number. What this means is that the original VicVac V (blue) changes to an intermediate VicVac V’’ (gray).
Importantly, note that the new VicVac V’’ is along the same direction as the original VicVac V, but scaled to be of a larger size.
B. Pure direction change
Next, lets design a compass action so that we can just rotate the new larger VicVac V’’ (gray), to get our desired final VicVac V’ (red).
Again, note that we dont change the size of the VicVac during this step.
Thus, the action of Magic Compass (MC) on VicVac (V) can be broken as combination of two actions:
- Pure Scaling, i.e. change the size while preserving the original orientation (blue V to gray V’’).
- Pure Rotation, i.e. change the orientation while preserving the size (gray V’’ to red V’).
“You got all of it Morpheus?”, asks Wincerind
“Yes, thanks. Consider writing a medium article to explain it to others too.”
“Good idea.”
Magic Compass = Matrices
In technical terms, change size = ‘scale’ and change direction = ‘rotate’.
Compass changes size and changes direction of VicVacs,
Matrices “scale” and “rotate” vectors.
This intuition is not trivial at all. The matrix magic via transformations of vectors to new ones is central to data science techniques.
I would recommend reading through the text above and solidifying the intuition again before proceeding to the next part. Note that for the previous example, the magical compass can be understood as the matrix
Eigen-VicVacs (Vectors)
Lets look again at our original magical compass acting on VicVacs composed of ibex and jackal parts.
If we take any arbitrary VicVac V, made up of a combination of ibex and jackal parts, and bring the magical compass close to it, we will get a new VicVac V’. The compass would first scale and then rotate, giving rise to a new V’.
But there are some special VicVacs (vectors), made in a special combination of ibex and jackal parts where the magical compass would first scale the VicVac but then lose the ability to rotate it.
These are our magical Eigen-VicVacs, the Eigenvectors.
Imagine a Vic-Vac from a family that has been bred in special farms with only pure ibex. Over millenia, the jackal parts have left this line of VicVacs and they can be regarded as almost pure ibex. Thus, we can denote them as (1i, 0j) VicVacs in our adventure book.
Now we take our compass to this farm and act on this (1i, 0j) VicVac. We can denote this action as:
Replacing the matrix form of the compass we get,
We find that the new VicVac is still 0 parts jackal but has 3x ibex parts!
The special VicVac (1i,0j) is called a Eigenvector of the matrix with an Eigen Value of 3. (More generally, any VicVac with 0 jackal parts is a Eigen-VicVac! )
Mathematical Note
“Eigenvectors” (like EigenVicVacs) are special vectors corresponding to a matrix M, which when acted by the matrix M are only scaled with no accompanying rotation. Intuitively, they do not allow the matrix M to weave its rotational action on them, but pose no challenge to it when it tries to change their size. This change of size of eigenvectors is given by their “Eigen-value”.
The matrix in our case has two eigenvalues 3 and 2 corresponding to eigenvectors (1i,0j) and (0i,1j). Note that any mutiple of an eigenvector is still an eigenvector, but its standard to write eigenvectors in the normalized form (i.e. form in which their euclidean distance from origin/“size” is equal to unity).
Hopefully, this clears your intuition about matrices and eigenvectors. I hope now when you pick up a math or engineering textbook, you will be reminded of the adventures of Wincerind!
