Intuitive Tutorial on Eigenvalue Decomposition in NumPy
Eigenvalue decomposition plays a central role in Mathematics. It is part of a set of factorization techniques, which aim to decompose (sometimes complex) mathematical objects as a product of several factors, usually smaller or simpler objects of the same kind. In this case, a matrix is decomposed into a more natural basis.
Why is it non-trivial?
Linear Algebra Reminder
For a given matrix (our data), an eigenpair is a couple (eigenvalue, eigenvector) that satisfy:
This basically means that the vector v is stable under the transformation induced by M, by a multiplicative factor lambda.
The interesting thing is that, collectively, all these eigenpairs form a basis of orthogonal vectors, which constitutes a more natural basis to look at the data given by M.
Computing an exact solution?
To obtain the exact solution, one can:
- Compute the following (equivalent to the definition above):
2. Solve the eigenvectors with Gaussian elimination.
Link with the characteristic polynomial
Recalling that the determinant above produces a polynomial whose roots are the eigenvalues, we can concur that:
- For small matrices, we can obtain the eigenvalues symbolically using the characteristic polynomial.
- As the rank of M increases, this method fails.
The Abel-Ruffini theorem
Abel Ruffini theorem explains the aforementioned limitation:
Any algorithm for finding the roots of polynomials for dimensions greater than 4 must either be infinite, or involve functions of greater complexity than elementary arithmetic operations and fractional powers.
Consequently, we should opt for numerical methods when dealing with larger matrices.
Iterative Algorithms
Iterative algorithms on the characteristic polynomial
Iterative algorithms that computes the roots of the characteristic polynomial exist, such as Newton’s method, but in general it is impractical to compute the characteristic polynomial and then apply these methods. One reason is that small round-off errors in the coefficients of the characteristic polynomial can lead to large errors in the eigenvalues and eigenvectors: the roots are an extremely ill-conditioned function of the coefficients.
Special matrices
Algorithms that exactly calculate eigenvalues in a finite number of steps only exist for a few special classes of matrices (symmetric, diagonal, tri-diagonal…).
Shifting redirection
Many iterative algorithms tend to compute eigenvalues one by one and not collectively. In this case, we don’t want our algorithm to detect twice an eigenpair. To avoid this situation, we transform the matrix M into:
This has the effect of “cancelling” the newly found eigenpair by reducing its impact.
Notably, Google uses this algorithm to compute PageRank (the foundation of the search engine).
The power method
For general matrices, algorithms are iterative, producing better approximate solutions with each iteration. A basic and accurate iterative method is the power method.
Let M be a n x n matrix whose eigenpairs we would like to find.
Compute one eigenpair
- Initialization: Start with any nonzero vector x and then iterate (with respect to the Frobenius norm):
- Stopping criterion: For a given error tolerance ε, if the difference between two consecutive iterations are almost identical, then the solution is our eigenvector.
- Computing the eigenvalue:
Putting it all together
Putting it all together consists of repeating n times:
- Compute the first eigenpair
- Apply shifting redirection
Therefore, we obtain the following algorithm:
Conclusion
To conclude briefly, for a given matrix, its eigenvectors are vectors that remains unchanged (by a multiplicative constant) when transformed by a matrix and that constitutes a natural basis to see our matrix.
When it comes to computing, we have no solutions for large matrices. Instead, we use iterative algorithms.
Go furher
Arnoldi iteration
For instance, by keeping not just the last vector in the sequence, but instead looking at the span of all the vectors in the sequence, one can get a better (faster converging) approximation for the eigenvector, and this idea is the basis of Arnoldi iteration.[6] Alternatively, the important QR algorithm is also based on a subtle transformation of a power method.[6]
Special Matrices
Depending on the type of the matrices, we can use more efficient algorithms. For the operating conditions of these algorithms, refer to this link.
