What Exactly Eigenvalues and Eigenvectors are?
You will learn the exact definition of eigenvalues and eigenvectors with a real-world example.
In machine learning and data science, we often use the Eigen theory. It is widely used in the data dimensionality reduction technique; PCA. Also, even Google’s search algorithm ‘PageRank’ is based on this concept. Once you read this article you will be able to understand what exactly the Eigen theory is and how it can be used to solve problems.
In linear algebra, we can represent an object with combinations of multiple vectors. When we apply any kind of transformation on an object, we shall observe its impact on all the vectors.
When a transformation is applied, some vectors end up staying in the same position. These vectors are called eigenvectors. The factor by which eigenvectors are scaled after the transformation is called eigenvalues.
Now let us understand this with an example of a square shown below(figure1). Although the square is made up of millions of vectors, for our ease of understanding we consider only 3 vectors r, s, v. Vector r, s, v are on plane1, plane3, plane2 respectively.
We apply transformation and let A be our transformation matrix.
Let’s have a very quick overview of what exactly the transformation matrix is.
When vector(s) are represented by a matrix, it must be read column-wise to identify vector(s).
Now, let’s see what exactly our transformation matrix A is doing to vector s. The x coordinate of vector s is 1 and the y-coordinate is also 1. (Make sure you are reading matrix A column-wise!)
When transformation matrix A is applied, x coordinate of s will move 1 unit in the x-direction and 0 unit in y-direction whereas y cordinate of s will move 0 unit in the x-direction and 2 unit in the y-direction.
So, we can see that after transformation coordinates of s are (1, 2).
The same transformation we apply to all the vector r, s, and v. Once the transformation is applied, our new coordinates are (0, 2), (1, 2), and (1, 0) respectively.
As you can see in figure 2, vector r and v are on the same plane (the plane is displayed with a dotted green line; that is plane1 and plane 2) as before whereas vector s is not on the same plane. plane3 is the original plane of vector s but when the transformation is applied it has changed.
Please note that when I say on the same plane, it means that those vectors are linearly dependent. Vectors are considered linearly independent if and only if c1 and c2 (equation 0)are equal to zero else it is linearly dependent.
For example, we saw vector s is not on the same plane after the transformation. Initially, coordinates of s were (1, 1) later it became (1, 2), let’s call it s’. Look at the formula mentioned below.
Therefore, we can conclude that vector r and v are our eigenvectors as they ended up staying in the same position(check figure 2). The length of vector v is the same so the eigenvalue for v is 1 and the length of vector r has been doubled following the transformation so the eigenvalue of r is 2.
Now, we will calculate all the possible eigenvectors and eigenvalues for the same example.
Our transformation matrix is A. let x be our set of eigenvectors and λ represents eigenvalue(s).
The relation between these 3 terms can be expressed with this equation (equation 1).
Here there are 2 possibilities, either x = 0 or (A-λI) = 0. x is a set of eigenvectors that we are supposed to calculate so it can not be zero. Hence, we will consider (A-λI) = 0.
To calculate eigenvectors and eigenvalues follow the steps mentioned below:
- With the help of equation 2, calculate all the possible value(s) of λ.
- Substitute the value of λ back to equation 1 and calculate x for each value of λ.
So, now we got two eigenvalues 1 and 2. For both cases, we will calculate x from equation 1.
For λ2, we calculated that (-x2, 0) = 0. It means that x2 must be zero, however, we do not care what y2 is; y2 can be anything (To verify, replace y2 with any random value, every time you get the same answer ‘0’!). Hence, (0, t) is eigenvectors’ set for eigenvalue 2, where t can be any number. Now check the vector r (Figure 2) from the previous example. For r, λ = 2 and eigenvector is (0, 2) which certainly verifies our above calculations.
The same explanation can be made for λ = 1 and can be verified by vector v from the previous example. (Figure 2)
Exercise:
Calculate eigenvectors and eigenvalues for shear operation where the transformation matrix is A.
Answer:
Set of eigenvectors are: (t, 0), for λ = 1. It has only one eigenvalue and one set of eigenvectors. Answer (t, 0) says that when transformation matrix T is applied, any vector on the x-axis is our eigenvector with 1 as an eigenvalue. In other words, whenever we apply transformation matrix T any vectors on the x-axis end up staying in the same position.
Let’s see graphical representation:
As you can see in Figure 4 after the transformation is applied, vector r and s have changed their respective plane; vector r moved from plane 2 to plane 3 and vector s moved from plane 3 to plane 4. In terms of linear algebra, r and r’, s and s’ are linearly independent. However, if you see vector v is on the same plane(Do check figure 4) so it is our eigenvector with eigenvalue of 1. Do check out this animation.
Real-World Example (Or Fantasy World Example!?!)
Let’s assume that we are in 2050 and as per Elon Musk’s Mars plan, there are 1 million people on Mars where birth and death rates are 15% and 12% respectively, and 1% of people moving back to the Earth(may be they don’t like Mars!). Also, there are 9 billion people on Earth where birth and death rates are 12% and 10% respectively and 5% of people are moving to Mars every year. Now the question is what would be the population of Earth and Mars in 2100?
Firstly, we need to put the data into a proper format so we will calculate the transformation matrix as mentioned below:
Where dE/dT means a change in Earth’s population per year and dM/dT means a change in Mars’s population per year.
Transformation matrix T represents vector E(Earth’s) and M(Mars’s). (Read column-wise)
Today(I mean in 2050!), the population of Earth and Mars is 9 billion and 1 million respectively. So vector P would be (9B, 1M). To calculate the population in 2100 we need to perform T⁵⁰ x P.
So in 2100, the population of Earth and Mars would be 5.6 billion and 26.4 billion respectively. Luckily in this problem, we dealt with just 2x2 matrix, however, in the real-world matrix can be so huge that let our systems run out space and time, if still, we use this method!
Now just imagine what if we had a diagonal matrix? Then finding nth power of the matrix would be so easy, right?
Here eigen theory comes into the picture. To find the nth power of matrix T we will use eigenvectors and eigenvalues. This method called Diagonalization of the matrix. Steps are mentioned below:
- Change the basis vectors of matrix T and let eigenvectors be a new basis vector or, in another word, moving vectors from the standard coordinate system (1,0), (0,1) to (x1, y1), (x2, y2). Call this matrix D.
- Let C be a diagonal matrix which is nothing but a matrix of eigenvectors.
- Calculate the nth power of matrix C (in our case n is 50).
- Again change the basis and bring back vectors to the standard coordinate system/ or whatever coordinate system was used before.
We have already prepared our transformation matrix T. Now we will calculate its eigenvalues and eigenvectors using the same method mentioned above. (However, calculation of eigenvectors is not mentioned here but you can directly calculate from here or review calculation 1.2 again). Calculations for matrix diagonalization is mentioned below:
Using this diagonalization, we have drastically reduced the number of computations and it just became so easy. Now just multiply T⁵⁰ with vector P (which denotes the populations). Results are here:
That is it! Now we know what would be the population of Earth and Mars in 2100. Numbers are just fascinating, right! Also, there might be some minor errors in the answers from both the methods due to round off so just ignore that.
Also, look at figure 5 to see how the population expands every year. (X-axis is for Earth’s population and Y-axis is for Mars’s population).
How to calculate in Python?
In python, using numpy library we can calculate eigenvalues and eigenvectors.
import numpy.linalg as la #numpy’s linear algebra library
import numpy as npT = np.array([[0.97, 0.01] #defining the transformatin matrix
[0.05, 1.02]])#calculating eigenvalues and eigenvectors of matrix TeigenValues, eigenVectors = la.eig(T)
print(eigenValues, eigenVectors)
In case of any doubts or queries feel free to contact me at 7mayurpshah@gmail.com
Peace!
