Image for post
Image for post
Image by Benjamin O. Tayo

Data Science

Mathematics of Principal Component Analysis with R Code Implementation

Theoretical foundations of principal component analysis (PCA) with R code implementation

I. Introduction

Generally, the larger the dataset the better, however, there is a tradeoff between the size of the dataset and computational time needed for training. It turns out that in very large datasets, there might be lots of redundancy in the features or lots of unimportant features in the dataset, and hence dimensionality reduction techniques could be used for selecting only a limited number of relevant features needed for training.

Principal Component Analysis (PCA) is a statistical method that is used for feature extraction. PCA is used for high-dimensional and highly correlated data. The basic idea of PCA is to transform the original space of features into the space of principal components, as shown in Figure 1 below:

Image for post
Image for post
Figure 1: PCA algorithm transforms from old to new feature space so as to remove feature correlation. Image by Benjamin O. Tayo

A PCA transformation achieves the following:

a) Reduce the number of features to be used in the final model by focusing only on the components accounting for the majority of the variance in the dataset.

b) Removes the correlation between features.

II. Mathematical Basis of PCA

Image for post
Image for post
Table 1. Features matrix with 4 variables and n observations.

To visualize the correlations between the features, we can generate a scatter plot, as shown in Figure 1. To quantify the degree of correlation between features, we can compute the covariance matrix using this equation:

Image for post
Image for post

In matrix form, the covariance matrix can be expressed as a 4 x 4 symmetric matrix:

Image for post
Image for post

This matrix can be diagonalized by performing a unitary transformation (PCA transformation) to obtain the following:

Image for post
Image for post

Since the trace of a matrix remains invariant under a unitary transformation, we observe that the sum of the eigenvalues of the diagonal matrix is equal to the total variance contained in features X1, X2, X3, and X4. Hence, we can define the following quantities:

Image for post
Image for post

Notice that when p = 4, the cumulative variance becomes equal to 1 as expected.

III. R Implementation of PCA

Let us look at the covariance matrix:

Image for post
Image for post
Table 2. Correlation matrix for the iris dataset.

Table 2 shows strong correlations between original features in the iris dataset. Figure 2 is a pairplot that shows scatter plots, density plots, and correlation coefficients between original features. Notice the strong correlations between original features.

Image for post
Image for post
Figure 2. Pairplot for iris dataset in original feature space.

Let us now examine the transformed covariance matrix:

Image for post
Image for post
Table 3. Covariance matrix in PCA space.

Table 3 shows zero correlations between transformed features. Figure 4 shows the pairplot in the PCA space. We see that the correlation between features has been removed.

Image for post
Image for post
Figure 3. Pairplot for iris dataset in PCA space.

Table 4 contains a summary of helpful indicators from a PCA calculation:

Image for post
Image for post
Table 4. Summary of useful indicators from PCA calculation.

Based on this summary, we see that 99.5 percent of the variance is contributed by the first three principal components (p = 3). This means that in the final model, the fourth principal component PC4 could be dropped since its contribution to the variance is negligible.

IV. Summary and Conclusion

Additional Data Science/Machine Learning Resources

Data Science Curriculum

Essential Maths Skills for Machine Learning

3 Best Data Science MOOC Specializations

5 Best Degrees for Getting into Data Science

5 reasons why you should begin your data science journey in 2020

Theoretical Foundations of Data Science — Should I Care or Simply Focus on Hands-on Skills?

Machine Learning Project Planning

How to Organize Your Data Science Project

Productivity Tools for Large-scale Data Science Projects

A Data Science Portfolio is More Valuable than a Resume

Data Science 101 — A Short Course on Medium Platform with R and Python Code Included

For questions and inquiries, please email me: benjaminobi@gmail.com

Towards AI

The Best of Tech, Science, and Engineering.

Sign up for Towards AI Newsletter

By Towards AI

Towards AI publishes the best of tech, science, and engineering. Subscribe to receive our updates right in your inbox. Interested in working with us? Please contact us → https://towardsai.net/contact Take a look

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

Benjamin Obi Tayo Ph.D.

Written by

Physicist, Data Science Educator, Writer. Interests: Data Science, Machine Learning, AI, Python & R, Predictive Analytics, Materials Sciences, Biophysics

Towards AI

Towards AI is the world’s leading multidisciplinary science publication. Towards AI publishes the best of tech, science, and engineering. Read by thought-leaders and decision-makers around the world.

Benjamin Obi Tayo Ph.D.

Written by

Physicist, Data Science Educator, Writer. Interests: Data Science, Machine Learning, AI, Python & R, Predictive Analytics, Materials Sciences, Biophysics

Towards AI

Towards AI is the world’s leading multidisciplinary science publication. Towards AI publishes the best of tech, science, and engineering. Read by thought-leaders and decision-makers around the world.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store