Eigenvectors and Eigenvalues and their use in Principal Component Analysis -Machine learning
Eigenvectors and eigenvalues have many important applications in different branches of computer science. The well-known examples are geometric transformations of 2D and 3D objects used in modelling software or Eigenfaces for face recognition, PCA (Principal Component Analysis) for dimensionality reduction in computer vision and machine learning in general.
In this article, let's discuss what are eigenvectors and eigenvalues and how they are used in the Principal component analysis.
Linear Transformation!
Let's think of a matrix A
and a vector B
when a linear transformation is applied to vector B with matrix A
you get another vector C
Now when we look at both vector B and C on a cartesian plane after a linear transformation, we notice both magnitude and direction of the vector B has changed. Therefore in linear transformation, a matrix can transform the magnitude and the direction of a vector sometimes into a lower or higher dimension.
Eigenvectors and Eigenvalues ?
Let's consider the above matrix A again
and a vector D
When a linear transformation is applied to vector D with matrix A
you get another vector E
Now when we look at both vector D and E on a cartesian plane after a linear transformation, we notice only the magnitude of the vector D has changed and not its direction. From this observation, we can define what an eigenvector and eigenvalue are.
An Eigenvector is a vector that when multiplied by a given transformation matrix is a scalar multiple of itself, and the eigenvalue is the scalar multiple.
here in our case vector D is our eigenvector and the eigenvalue is 2 as vector D had scaled to vector E by a factor of 2.
In this article, we won't be focusing on how to calculate these eigenvectors and eigenvalues. will provide references to these tutorials at the end of the article.
Principal Component Analysis
Principal Component Analysis, or PCA, is a dimensionality-reduction method that is often used to reduce the dimensionality of large data sets, by transforming a large set of variables into a smaller one that still contains most of the information in the large set.
Reducing the number of variables of a data set naturally comes at the expense of accuracy, but the trick in dimensionality reduction is to trade a little accuracy for simplicity. Because smaller data sets are easier to explore and visualize and make analyzing data much easier and faster for machine learning algorithms without extraneous variables to process.
Now let's understand how the principal component is determined using eigenvectors and their corresponding eigenvalues for the below-sampled data from a two-dimensional Gaussian distribution.
After collecting the data samples we need to understand how the variables of the input data set are varying from the mean with respect to each other, or in other words, to see if there is any relationship between them. Because sometimes, variables are highly correlated in such a way that they contain redundant information. So, in order to identify these correlations, we compute the covariance matrix.
A covariance matrix is a symmetric matrix that expresses how each of the variables in the sample data relates to each other.
Now we need to find a new axis for the data such that we can represent every two-dimensional point with values (x,y) by using a one-dimensional scalar r, value r is the projection of the point (x,y) onto the new axis, to achieve this we need to calculate the eigenvectors and the eigenvalues of the covariance matrix.
Geometrically speaking, principal components represent the directions of the data that explain a maximal amount of variance, that is to say, the lines that capture most information of the data.
Organizing information in principal components this way will allow reducing dimensionality without losing much information, and discarding the components with low information and considering the remaining components as your new variables.
