Google Summer Of Code - Part 2 (Deep AutoEncoders)

Akshay Vashistha
Aug 26, 2017 · 4 min read

Hello guys! In this story I would like to discuss about Deep AutoEncoders.

The best definition is given by Stanford :

An autoencoder neural network is an unsupervised learning algorithm that applies backpropagation, setting the target values to be equal to the inputs. I.e., it uses y(i)=x(i).

This is a simple AutoEncoder. An AutoEncoder comprises of Visible Units, Hidden Units and Reconstructed Units. These can be considered as different layers. Consider the input vector as x

f(x) = s(Wx+b) ……..…………………………………….………………..(i)

is used to encode the input into the hidden units where W is the weights matrix associated within the first two layers, b is the biases vector associated with visible units and s is the sigmoid activation function. This encodes the input from lets say 6 visible units to 3 hidden units or something like that. Now we have to reconstruct the output from these hidden units this is the decoding step where input has to be decoded from the encoded version.

g(y) = s(W′y+b′) …………………………………………………………..(ii)

here y is the hidden vector. We use concept of tied weights to get better output i.e between layer 2 and 3 we used the transpose of weights between layer 1 and 2. So W′ is the transpose weights matrix, b is the biases vector associated with hidden units and s is sigmoid activation fn.

This was a simple forward pass. We then use backpropagation to train the Network. The hidden units form a compressed trained version of inputs and the recontructed layer reconstructs the original input.

Using Corrupted Inputs

To improve what the network learns, we train the network not on original inputs but on there corrupted version. We mask the input values randomly according to the corruptionLevel like we can mask 10% or 30% of inputs to other values or simply a zero and train the network.

As shown in the figure, the network tries to learn and get back the original units from the corrupted version. Thus this type of version can be used easily for Anamoly Detection and getting correct i/p. Maybe we can also use it to get correct signals when there is some corruption in network.

This is how we train a corrupted AutoEncoder. First we corrupt the input vector from x to tilde x. Then this tilde x is used as an input and we get encoded vector through it(using (i)). Then using (ii) we can get the reconstructed version of input. The weights and biases are then updated. But keep in mind, we calculate the difference between reconstructed vector(lets say z) and input vector x not tilde x. The error is than backpropagated and we update weights and biases.

Deep Network

This is a Deep Network of AutoEncoders. Here we take an input vector, Train the layers to lets say N no of epochs. The then generated encoded version i.e the version of encoded layers after training to N epochs is passes as an input to next layer. This continues to form a Deep Network of AutoEncoders. This successive training of AutoEncoders is called PreTrain(). This can be used to extract various trained features after every layer.

This type of Network can be of great use to get optimized results. We can add any type of classification Layer in the end to get more accurate results. This is because we are training the classifier not on inputs but on trained features. This is termed as FineTuning where we Finetune the last layer output to classify. For eg in the image above a softmax classifier is used to get the output.

In the next post I will tell about how I implemented the same for TMVA.

Cheers!

)
Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade