Linear Algebra for Machine Learning Part 5— Singular Value Decomposition (SVD) and Matrix Pseudo Inverse

Singular Value Decomposition (SVD) :

Matrix Pseudo Inverse:

One drawback of matrix inverse is that it is defined only for square matrices. With matrix pseudo inverse, we overcome this limitation. From Wikipedia —

A pseudoinverse A+ of a matrix A is a generalization of the inverse matrix. The most widely known type of matrix pseudoinverse is the Moore–Penrose inverse.

…The pseudoinverse is defined and unique for all matrices whose entries are real or complex numbers. It can be computed using the singular value decomposition.

Use :

• A common use of the pseudoinverse is to compute a ‘best fit’ or least squares solution (also called the “Normal Equation”) to a system of linear equations. See in the references.
• Used in PCA (Principal Component Analysis)

References — SVD :

• https://www.cs.cmu.edu/~venkatg/teaching/CStheory-infoage/book-chapter-4.pdf
• http://mathworld.wolfram.com/LeastSquaresFitting.html
• http://andrew.gibiansky.com/blog/mathematics/cool-linear-algebra-singular-value-decomposition/

References — Matrix Pseudo Inverse :

• http://buzzard.ups.edu/courses/2014spring/420projects/math420-UPS-spring-2014-macausland-pseudo-inverse.pdf
• http://people.revoledu.com/kardi/tutorial/LinearAlgebra/MatrixGeneralizedInverse.html
• http://mathonline.wikidot.com/inverse-of-diagonal-matrices
• https://en.wikipedia.org/wiki/Moore%E2%80%93Penrose_inverse
• Best fit or least squares solution to a system of linear equations — http://cs229.stanford.edu/section/cs229-prob.pdf (Page - 25)

Please give a clap if you have liked the article, comment if you have any doubt/suggestion/correction.

All parts in this series —

Part 1 — Building Blocks

Part 2 — Matrices

Part 3 — System of Linear Equations

Part 4 — Linear Transformation and Eigendecomposition(EVD)

Part 5 — Singular Value Decomposition (SVD) and Matrix Pseudoinverse

Part 6 — Dimensionality Reduction and PCA