Inside AI

Unveiling the Hidden Layers of Neural Networks

A guide to visualizing the black-box of Deep Learning

Published in

SFU Professional Computer Science

11 min readFeb 4, 2020

Authored by Pallavi Bharadwaj, Inderpartap Cheema, Najeeb Qazi, Mrinal Gosain
This blog is written and maintained by students in the Professional Master’s Program in the School of Computing Science at Simon Fraser University as part of their course credit. To learn more about this unique program, please visit {sfu.ca/computing/pmp}.

Networks that get only smarter

Connections from one neuron to another in the brain’s cortex have inspired the creation of algorithms that mimic these intricate links. The learning approach of a neural network can be understood, for the sake of simplicity, to be analogous to the manner in which newborns learn to recognize their parents- by observing and listening to them over a period of time. Once a Neural Network trains on a considerable size of the required dataset, the network uses its knowledge- ‘weights’ to be precise — to identify or recognize patterns on an unknown data sample.

Let’s consider the task of identifying faces, the network would begin analyzing individual pixels of an image from the input layer. Following the input layer, the “hidden” layers iteratively learn to identify geometric shapes and features that are distinctive to a particular face like the eyes, lips, scars, etc that make up the entire image. In the final output layer, it makes a well-educated guess about whom the face belongs to, from the probability information it has calculated.

Evidently, these hidden layers play a vital role in eventually breaking down input into valuable information. Each layer from input until the output handles increasingly complex information, and it is often said that — the hidden layers — as their name suggests — are shrouded in mystery, but are they?

The Neural Net Tank urban legend

AI folklore tells a story about a neural network that was trained with an intention to detect tanks but instead, it learned to detect the time of the day

The US Army wanted to use neural networks to automatically detect camouflaged enemy tanks. The researchers trained a neural network on two sets, one with 50 photos of camouflaged tanks amid trees, and the other with 50 photos of trees without tanks. The researchers then took another set of 100 images and tested on them. The neural network classified all remaining photos correctly. Success confirmed! The researchers handed over their work to the Pentagon, which soon handed it back, complaining that further tests completely failed on the neural network. It turned out that in the dataset that was used for training, images of camouflaged tanks had been taken on cloudy days, while the images of the plain forest had been taken on sunny days. The neural network had learned to distinguish cloudy days from sunny days, instead of learning to identify camouflaged tanks.

Regardless of the authenticity of the story, it highlights an important problem of ‘data bias’ in deep learning, but it also made some people believe that they could never know what a neural network is learning until they had a look at the final output. Even if the result is right for the given data, it is important to know the basis of how the network arrived at an outcome, which makes it pertinent to understand the workings of the hidden layers of the network.

What’s going on inside?

We go back to the basic building blocks of our networks — The Neurons. Layers and layers of neurons make up our complex state of the art neural networks. But the question is what does each neuron learn? Is it possible to capture the image of the network at any given time and see how each neuron behaves? As the saying goes — “A picture is worth a thousand words”, the need for visualizing arises at this stage.

In a recent project, Daniel Smilkov, one of the co-authors of Tensorflow.js and Shan Carter, a member of the Google Brain team, created a neural network playground, which aims to visualize the hidden layers by allowing users to interact and experiment with them.

Interestingly, as we build better models for machine to learn, we may end up revealing new information about how our own brains work. Playing with and visualizing the hidden layers is a great way to begin this process while also making the concept of deep learning accessible to a wider audience.

Once you have had fun and are now bored with the Tensorflow Playground, join us back to learn how to build your own visualizations.

Diving deeper into Deep Learning

Let us introduce you to the Keras visualization toolkit (Keras-vis), a high-level library for visualizing and debugging a trained Neural Network.

We are using pre-trained weights of the VGG16 network trained on a subset of the ImageNet dataset consisting of 1.2 million images hand-annotated as belonging to one of the 1000 predefined classes.

Model: "vgg16"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_2 (InputLayer)         (None, 224, 224, 3)       0         
_________________________________________________________________
block1_conv1 (Conv2D)        (None, 224, 224, 64)      1792      
_________________________________________________________________
block1_conv2 (Conv2D)        (None, 224, 224, 64)      36928     
_________________________________________________________________
block1_pool (MaxPooling2D)   (None, 112, 112, 64)      0         
_________________________________________________________________
block2_conv1 (Conv2D)        (None, 112, 112, 128)     73856     
_________________________________________________________________
block2_conv2 (Conv2D)        (None, 112, 112, 128)     147584    
_________________________________________________________________
block2_pool (MaxPooling2D)   (None, 56, 56, 128)       0         
_________________________________________________________________
block3_conv1 (Conv2D)        (None, 56, 56, 256)       295168    
_________________________________________________________________
block3_conv2 (Conv2D)        (None, 56, 56, 256)       590080    
_________________________________________________________________
block3_conv3 (Conv2D)        (None, 56, 56, 256)       590080    
_________________________________________________________________
block3_pool (MaxPooling2D)   (None, 28, 28, 256)       0         
_________________________________________________________________
block4_conv1 (Conv2D)        (None, 28, 28, 512)       1180160   
_________________________________________________________________
block4_conv2 (Conv2D)        (None, 28, 28, 512)       2359808   
_________________________________________________________________
block4_conv3 (Conv2D)        (None, 28, 28, 512)       2359808   
_________________________________________________________________
block4_pool (MaxPooling2D)   (None, 14, 14, 512)       0         
_________________________________________________________________
block5_conv1 (Conv2D)        (None, 14, 14, 512)       2359808   
_________________________________________________________________
block5_conv2 (Conv2D)        (None, 14, 14, 512)       2359808   
_________________________________________________________________
block5_conv3 (Conv2D)        (None, 14, 14, 512)       2359808   
_________________________________________________________________
block5_pool (MaxPooling2D)   (None, 7, 7, 512)         0         
_________________________________________________________________
flatten (Flatten)            (None, 25088)             0         
_________________________________________________________________
fc1 (Dense)                  (None, 4096)              102764544 
_________________________________________________________________
fc2 (Dense)                  (None, 4096)              16781312  
_________________________________________________________________
predictions (Dense)          (None, 1000)              4097000   
=================================================================
Total params: 138,357,544
Trainable params: 138,357,544
Non-trainable params: 0
_________________________________________________________________

This gives us an insight into the kind of layers used, filter size, trainable parameters, etc. Note that the pooling layer has 0 parameters since the layer is not trainable. The MaxPooling layer reduces the size of the input by picking the maximum value in the window, and this process does not involve any weight updates.

If we are to visualize, say just the output layer of an image classification problem, as an image, then we would want to convert the activation function at the output layer to Linear instead of a Softmax. It comes in handy to know a way of retrieving configuration parameters and weights associated with each layer of the model.

{'activation': 'relu',
 'activity_regularizer': None,
 'bias_constraint': None,
 'bias_initializer': {'class_name': 'Zeros', 'config': {}},
 'bias_regularizer': None,
 'data_format': 'channels_last',
 'dilation_rate': (1, 1),
 'dtype': 'float32',
 'filters': 128,
 'kernel_constraint': None,
 'kernel_initializer': {
     'class_name': 'VarianceScaling',
     'config': {'distribution': 'uniform',
                'mode': 'fan_avg',
                'scale': 1.0,
                'seed': None}},
 'kernel_regularizer': None,
 'kernel_size': (3, 3),
 'name': 'block2_conv1',
 'padding': 'same',
 'strides': (1, 1),
 'trainable': True,
 'use_bias': True}

This gives us some valuable information like the activation functions used at each layer, whether the layer is trainable or not, filter size, type of regularizer used, and so on.

Activation Maximization: Visualizing what a model expects.

This utility generates the input that maximizes the activation of the specified filter at the given layer. Here, we try to analyze what the model expects to classify an input image as a Black Bear.

As humans, we know that the common characteristics of Bears are large bodies with stocky legs, long snouts, small rounded ears, shaggy hair, plantigrade paws with five non-retractile claws, and short tails.

There are multiple levels of decisions to be made here to finally conclude with the class. We need to know that this is a picture of an animal and that it is a bear, and then we narrow it down to the class of bear (polar/black/brown, etc) based on its salient features.

We know that black bears are found in the woods and polar bears live in snowy regions. As humans, we still correctly identify an image of a black bear even on a white background. This is also what a well-trained network is expected to learn progressively across the multiple layers. We do not want the network to distinguish green and white backgrounds but classify the types of bears based on their features. We don’t want to end up as that urban legend, do we?

Images of Black Bear (left) and Polar Bear (Right)

Let us try and visualize what the model expects as an input, to classify it as a black bear. To classify an image as a black bear, we need the 295th index of the output layer to be activated, since this is the index corresponding to a ‘black bear’. So we go backward constructing an input image by activating this node of the output layer.

(Each node of the output layer corresponds to a class of images. If you wish to view all the classes and indexes of the ImageNet dataset, you can do so here.)

Activation Maximization plots for Black Bear (left) and Polar Bear (right)

We see from the plots above that at the final layer, the model is looking for most of the particular features of a bear we mentioned above. Minute feature differences of each of the types are learned progressively in the training process. We can now be confident that the network is learning the right set of features in order to call a bear, a bear!

The Process — Layerwise Output Visualization

You might be wondering how each hidden layer contributes to the final result at the output layer. This is particularly important in order to fine-tune the model to get expected results. How? Gaining an understanding of which layer highlights what set of features enables us to use skip connections to ignore those features when required.

Let’s consider the previous example of the classification of bears. We now try to understand how the network is guided throughout the process of classification. We try to extract the output of a couple of hidden layers (block1_conv1, block2_conv1, block3_conv1, block4_conv1) of the network and plot the image at each layer.

Layer-wise output for black bear (left) and polar bear (right)

Note that the initial layers are learning low-level features like edges and shapes. These layers learn the gist of the input. In subsequent layers, the network tries to expose more and more obscure patterns of the image like the legs, ears, eyes, color, etc. This is why we would say, the deeper the network, the better it learns.

The mysterious hidden layers of the network progressively capture the most intricate features of the input.

Saliency Maps

Saliency in images refers to unique features (pixels, resolution) of an image in the context of visual processing. They represent the most visually alluring locations of the image. Saliency maps are a topographical representation of them. This visualization technique allows us to recognize the importance of each pixel in generating the output. This technique was first introduced in the paper: Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps.

Saliency maps produce an attention heatmap of the seed input that activates a class. Let us see what this is like with an example.

A brown dog with a red collar — That’s a cute dog. Isn’t it?

What part of the image do you think the network should be concerned about in the image above? Certainly, it’s the dog on the right portion of the image. The rest of the image pixels are to be ignored in the image to correctly classify this image as a picture of a dog. Now, how do we know if our network is focusing on this part? This is where saliency maps play an important role.

Saliency maps calculate the effect of every pixel on the result. This involves computing the gradient of the output with respect to every pixel on the input image. A positive gradient indicates that a change in pixel value increases the output value.

Saliency Map — Heatmap of the pixel attention

As we discussed above, the model is focusing on the face of the dog in the image. Since the size of the gradients is equivalent to the size of the image (calculated on every pixel), this provides us an intuition of attention.

Occlusion Maps

Saliency maps are useful for finding the unique features in an image, Occlusion maps, on the other hand, are useful to find out which part of the image, is the most important for the model.

Occlusion, in English, means to “hide or block” something. This is exactly how this works. Some parts of the image are masked off, or “occluded” and the probability of the class is measured. If it decreases, that portion of the image is important, otherwise, it isn’t.

It can be better understood with an image of this little munchkin monster.

We start by loading this image and then plotting it. The next step would be to generate a heatmap of probabilities, by masking different portions of the image.

The map is now converted into a grayscale mask using the standardized probabilities and in the end, is imposed on our image.

Masked final image (left) and the grayscale mask (right)

With this, we can see which parts of the image are the most important for the model to classify the image. This provides much greater insight into understanding the reasons for misclassification.