Google Summer Of Code - Part 3 (Integrating AutoEncoder in TMVA)

Hey Guys! So in this story I would like to tell how did I actually integrated Deep AutoEncoders in TMVA.

So as told we had to implement the Deep Learning Algorithms over DNNs built by Simon Pfreundschuh, a GSOC 2016 student. As we were three students(Me, Vladimir and Saurav) selected for Deep Learning Team. There were three Methods to be booked, the basic layout was same for all the layers but the modules differed. Like there were three DeepNet classes. So it was like the related things were being developed in mutually exclusive manner. I almost developed the Deep Autoencoder layers and DeepNet within one week after First Evaluation. Vladimir then proposed a very good idea and we decided to generalize all the layers. There should be a single DeepNet for all the Algorithms. There should be a general layer with some virtual functions that need to be defined by every branch that extends general layer.

There are Tensors introduced as the type of input, previously it was Matrices. So the new DeepNet looks like

Deep Learning Module — TMVA

This new DeepNet is much more flexible. It can add any type of layers like we can use unsupervised Autoencoders and then classify the outputs using CNNs or maybe a Fully Connected Layer.

So now after first evaluations we migrated to a new version of Layers. This took some time. We needed a common DeepNet that satisfies all the layers. A general Layers that fulfills everything in subsequent layers. So now there is a common structure.

All the AutoEncoder Layers can be viewed here. The DeepNet can be viewed here. In the library we will have CNN’s and RNN’s functions and layer pointers also.

Structure of Layers

Corruption Layer

Compression Layer

Reconstruction Layer

Logistic Regression Layer

The pointers to Layers are added in the fLayers vector in the DeepNet. Moreover there are PreTrain(), FineTune(), PredictEncodedOutput() and PredictDecodedOutput() in the DeepNet which are exclusive for AutoEncoders. The DeepNet suited for AutoEncoders can be viewed here. All my commits can be viewed here and here. As we were three students selected to create a common module, we were working on a common repo so merge will be done when everything finishes. Please read all my GSOC stories to get full insight of my work.

In my next story I will discuss about how I booked this for use in TMVA and the challenges faced.

Cheers!