What are Eigenvalues and Eigenvectors?

A must-know concept for Machine Learning

Farhad Malik
Jan 6 · 9 min read

Eigenvalues and eigenvectors form the basics of computing and mathematics

Quick Introduction

Eigenvector —Every vector (list of numbers) has a direction when it is plotted on a XY chart. Eigenvectors are those vectors when a linear transformation (such as multiplying it to a scalar) is performed on them then their direction does not change. This attribute of Eigenvectors make them very valuable as I will explain in this article.

Photo by Helloquence on Unsplash

Article Structure

  • Firstly I will give a brief introduction of eigenvectors and eigenvalues.
  • Then I will explain how eigenvectors and eigenvalues are calculated but I will explain them from the foundation of matrix multiplication and addition so that we all can understand them thoroughly.
  • I will then present a working example and we will calculate the eigenvectors and eigenvalues together
  • Lastly I will outline how we can compute the eigenvectors and eigenvalues in Python.

It will build our confidence in dimension reduction techniques which are crucial to understand in data science

You are probably wondering why eigenvalues and eigenvectors


Introducing Eigenvalues and Eigenvectors — Where are they used?

When we are building forecasting models that are trained on images, sound and/or textual contents then the input feature sets can end up having a large set of features. It is also difficult to understand and visualize data with more than 3 dimensions. As a result, we often use one-hot encoding to transform values in textual features to separate numerical columns which can end up taking a large amount of space on disk.

Eigenvalues and Eigenvectors are the key tools to use in those scenarios

Eigenvalues and Eigenvectors have their importance in linear differential equations where you want to find a rate of change or when you want to maintain relationships between two variables.

Think of eigenvalues and eigenvectors as providing Summary of a large matrix

We can represent a large set of information in a matrix. One eigenvalue and eigenvector is used to capture key information that is stored in a large matrix. Performing computations on a large matrix is a very slow process. To elaborate, one of the key methodologies to improve efficiency in computational intensive tasks is to reduce the dimensions after ensuring most of the key information is maintained.

It is now apparent that Eigenvalues and Eigenvectors are one of core concepts to understand in data science. Hence this article is dedicated to them.

Practical usecases of Eigenvectors and Eigenvalues:

They are used to reduce dimension space. If you want to forecast a financial variable e.g.interest rates then you will gather data for variables that interest rate is dependent on. Then you will join the variables to create a matrix. You might also be loading textual information and converting it to vectors.

Photo by Joel Filipe on Unsplash

More usecases of Eigenvalues and Eigenvectors

Occasionally we gather data that contains a large amount of noise. Finding important or meaningful patterns within the data can be extremely difficult. Eigenvectors and eigenvalues can be used to construct spectral clustering.

What are Eigenvalues and Eigenvectors?

Eigenvectors are used to make linear transformation understandable. Think of eigenvectors as stretching/compressing a X-Y line chart without changing its direction.

Let’s quickly recap and refresh how matrix multiplication and addition works before we take a deep dive

Matrix Addition

Matrix addition is simply achieved by taking each element of a matrix and adding it together as shown below:

Multiplying Scalar With A Matrix

Multiplying Matrix by a scalar is as straight forward as multiplying each element by the scalar:

Matrices Multiplication

Matrices multiplication is achieved by multiplying and then summing matching members of the two matrices. The image below illustrates how we can multiple a 3 by 3 and a 3 by 1 matrix together:

That’s all the Maths which we need to know for the moment

Let’s utilise the knowledge above to understand eigenvectors and eigenvalues:

Although we don’t have to calculate the Eigenvalues and Eigenvectors by hand every time but it is important to understand the inner workings to be able to confidently use the algorithms.

Key Concepts: Let’s go over the following bullet points before we calculate Eigenvalues and Eigenvectors

Eigenvalues and Eigenvectors have following components:

  • A square matrix is the one which has a size of n, implying that X and Y are equal.
  • Square matrix is represented as A. This is an example of a square matrix
  • Therefore, Eigenvector should be a non-null vector
  • Now Eigenvalues: We are required to find a number of values, known as eigenvalues such that
A * x - Lambda * x = 0
  • Last component: Identity Matrix. A square matrix which has 1 as diagonal and all other elements are 0 is known as an identity matrix. Identity matrix is represented as I:
A * x - Lambda * x = 0
x * (A - Lambda * I) = 0
Determinant(A - Lambda * I) = 0

How do I calculate Eigenvalue?

For a matrix A of size n, find Eigenvalues of size n.

 A * Eigenvector — Eigenvalue * EigenVector = 0

Find Lambda Such that Determinant(A — Lambda * I) = 0

Based on the concepts learnt above:

How do I calculate Eigenvector?

Once we have calculated eigenvalues, we can calculate the Eigenvectors of the matrix A by using Gaussian Elimination. Gaussian elimination is about converting the matrix to row echelon form. Finally it is about solving the linear system by back substitution.

x * (A - Lambda * I) = 0

Now that we have the key, it is the time to compute the Eigenvalues and Eigenvectors together with me

Photo by CMDR Shane on Unsplash

Let’s calculate Eigenvalue and Eigenvector together

If there are any doubts then do inform me.

Now that we have computed Eigenvalues, let’s calculate Eigenvectors together

Take the first Eigenvalue (Lambda) and substitute the eigenvalue into following equation:

x * (A - Lambda * I) = 0

Calculate Eigenvalues and Eigenvectors In Python

Although we don’t have to calculate the Eigenvalues and Eigenvectors by hand but it is important to understand the inner workings to be able to confidently use the algorithms. Furthermore, It is very straightforward to calculate eigenvalues and eigenvectors in Python.

from numpy import linalg as LAinput = np.array([[2,-1],[4,3]])w, v = LA.eig(input)

Summary

This article explained one of the key areas of machine learning. It started by giving a brief introduction of eigenvectors and eigenvalues.

FinTechExplained

This blog aims to bridge the gap between technologists, mathematicians and financial experts and helps them understand how fundamental concepts work within each field. Articles

Farhad Malik

Written by

Explaining complex mathematical, financial and technological concepts in simple terms. Contact: FarhadMalik84@googlemail.com

FinTechExplained

This blog aims to bridge the gap between technologists, mathematicians and financial experts and helps them understand how fundamental concepts work within each field. Articles