Matrix Inversion and Meaning
How a matrix inversion blew my mind.
If you are into data science or machine learning, then you must know about linear algebra. As Professor Gilbert Strang puts it in his latest book [1]:
Linear algebra and probability/statistics and optimization are the mathematical pillars of machine learning.
If you ever built a neural network from scratch, you probably used matrices or tensors and their mathematical properties to implement the forward or backward passes. Reducing the dimensionality of a data set also involves linear algebra theory, specifically through principal component analysis.
That is one of the reasons that I made it a personal goal to keep on updating my knowledge of linear algebra, mostly through books and online resources. I also found out that linear algebra is of great value when using discrete-time signal analysis, and that’s where I discovered the matrices that blew my mind.
Finite-length discrete-time signals can be represented as real-valued vectors [2]. To transform these signals, some special matrices can be used. Take for instance integration, the operation you could use to transform a unit impulse into a unit step. A unit impulse u and the unit step s are encoded this way:
Integration of one signal into the other can be performed by multiplying a special matrix S with u:
Now, what happens when you invert the matrix S? You get another matrix, let us call it D, that looks like this:
Looking at this equality from the linear algebra perspective, one would say that S undoes what D does, besides the obvious SD=I, where I is the identity matrix — the multiplication neutral element.
But undoing integration is differentiation, is it not? Let’s see:
As expected, the inverse of the integration matrix yields a differentiation matrix (left differences, to be precise). So what’s so special about this?
I believe that this demonstrates how mathematics is self-consistent and full of meaning. The inverse of a matrix that adds produces a matrix that subtracts! Inversion is an operation that is, in a sense, akin to division. It’s like calculating the reciprocal of a scalar, but for matrices.
The inverse of summation is differentiation. Literally.
References
[1] Strang, G. Linear Algebra and Learning from Data. 2019, Wellesley-Cambridge Press
[2] Prandoni, P. and Vetterli, M. Signal Processing for Communications. 2008, EPFL Press
