Autoencoder and PCA for Dimensionality reduction on MNIST Dataset…(with code).

CHAUDHARI AMOL MOHAMMAD
Nov 7 · 4 min read

In this blog we will see

  1. What is Autoencoder?
  2. How to Code it in Keras?
  3. Then we will compare PCA and Autoencoder result.

Let’s get started…

Autoenoder is an unsupervised learning where neural network are subject to task of Representation learning. In Simple word it imposed a bottleneck in network.(Bottleneck forces a compressed knowledge representation of input).


Note:

Here input and output are same in practical application,when Autoencoder is fully trained we remove decoder part and use encoder part in our model.

What we Expect from Autoencoder:

  1. Sensitive enough to input for accurate reconstruction.
  2. Insensitive enough that it does not memorize or overfit the training data.

Implementation of Autoencoder in keras:

Loading MNIST dataset:

Normalizing and reshaping our data:

Now we tried to compress our dimension with neural network….we are ready to build our simplest Autoencoder .

Our first Autoencoder consist of single fully connected layer as encoder and decoder.

Note:

In simple words what we are trying to do is, each image in our MNIST dataset is of dimension (28 X 28),already we reshape it into (784 X 1). Now we implement single fully connected layer network which we will train to compress the dimension into (32 X 1).

Model:784–32–784

Now we train our model with just 5 epoch.

Result :

With only 5 epoch Autoencoder reconstructed unseen test data significantly.

Why to limit ourself….with just single fully connected layer…..Deep Autoencoder reconstruct data way more significantly than single layer.

Next we tried to build deep Autoencoder and trained it for 5 epoch.

Model:784–128–64–32–64–128–784

Result: we can see that reconstructed data is more clear than previous one.

PCA :

Principal Component Analysis is an classic machine learning technique use for dimension reduction.

Now we try to do same thing using PCA .

Visualizing Result:

We use same dimension here to compress and then again decompress it and plot it ,it is clearly visible that result from Autoencoder is way better.

Next we did same thing ,but here we use 100 principal component to reduce dimension and plotted the result.

Now Question arises when to use PCA and when to use Autoencoder.

Note:

  1. PCA is an linear transformation where given set of data approximated by straight line.
  2. While Autoencoder can learn non linearity structure which is present in data .
  3. If we don’t impose any non linearity in network then Autoencoder and PCA are almost similar.

Application of Autoencoder:

There are various type of Autoencoder :

  1. Denoising Autoencoder.
  2. Sparse Autoencoder.
  3. Convolution Autoencoder.
  4. Variational Autoencoder.

Now we will see how to use Convolution Autoencoder for image denoising,for other application you can refer: https://iq.opengenus.org/types-of-autoencoder/

Convolution Autoencoder for image denoising:

  1. We added some noise to input data:

2. Implement model:

Note:

The encoder will consist in a stack of Conv2D and Maxpooling2D layers (max pooling being used for spatial down-sampling), while the decoder will consist in a stack of Conv2Dand Upsampling2D layers.

Model

Result:

Code:

https://github.com/Amol2709/AE-vs-PCA

Refrence:

  1. https://www.cs.toronto.edu/~hinton/science.pdf
  2. https://blog.keras.io/building-autoencoders-in-keras.html

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade