Visualizing Deep Learning with t-SNE (Tutorial and Video)

Convolutional neural networks have drastically changed the kinds of tasks that can be learned by computers, especially within the field of computer vision and classification. One of the issues with these networks however is that they are often difficult to visualize and develop an intuition of what exactly they are doing. Especially when first learning how to create models, their output, while often impressive (99% accuracy, wow!) can be mysterious.

One tool developed to help allow for visualization of these neural networks is t-SNE. By performing what is essentially dimensionality reduction, similar to principal components analysis, t-SNE is able to take the high-dimensional spaces that neural networks use and convert them into lower-dimensional spaces that humans can actually make sense of. I have begun incorporating this technique into my own work, and decided to put together an iPython notebook to walk through those interested in quickly getting visual representations of their networks as well.

Like my last tutorial, I am using the MNIST dataset as the training set. This time however, I am using Google’s Tensorflow framework to more easily train and model the network, as well as get the representations we want. Google has a series of helpful tutorials for those who are new to Tensorflow. In the notebook below, I extend one of their basic tutorials to show how we can get visuals on the network's representations as it learns.

I also decided to take this one step further, and put together a small Unity3d program that would create a 3D visualization of the network learning it’s representation of the dataset. Below you can watch as the network goes from a fuzzy representation to being able to separate the test examples into discreet groups.

UPDATE: I just released a new app that will allow you to visualize your own data in 3D using t-SNE. Read about and download it here.