Map your brain with Deep Learning

Animation by Alex Norton.

Here is a practical guide on how to map your brain — make sure to read until the end before you start.

The first step is to cut your brain into very thin slices and take a picture of each of them. How thin? At least 40 nanometers thin.

The next step is to identify each neuron within the slices with a unique color. By doing that, we can see how these neurons are connected to each other (really useful if you are trying to stimulate your brain in a computer).

Animation by Daniel Berger.

Given that you have 80 billion neurons, you might want to automate this process. Let me show you how.

In the case you want to experiment with someone else’s brain first you can download images from this challenge. Or clone this repo, which comes with some other convenient stuff.

This will download a three files:

  • train-input.h5 : a 3d array containing the electron microscopy images.
  • train-labels.h5: the corresponding 3d array containing an unique number of each neurite.
  • test-input.h5: another set of 3d electron microscopy images.
train-input.h5 (left), test-input.h5(right), image by Kisuk Lee.

Why don’t we take a look at the data first?

This will open a new tab in your web browser showing the input and labels superposed.

Training a network to output a unique number for each neuron is hard, so we will transform the labels into an affinity representation. For any two consecutive pixels in the “x” dimension, if they belong to the same neuron, their affinity is 1; if they belong to different neurons, their affinity is 0. If both pixels belong to the boundary, the convention is to set the affinity as 0. We will do this for the three axes.

x affinities. Image by Kisuk Lee.

It is also possible to visualize all three (x,y,z) affinities simultaneously, by making x: red, y:green, and z:blue(we will ignore the z affinities for now, to only training a 2d convet).

By hovering over a layers in the top left is possible to change the opacity.

Time to train a neural network to predict the affinities from the input image. We will use a sliding window convnet from this paper called N4. An sliding window convnet gets a patch of input image and has two numerical outputs which corresponds to the x,y affinities of the pixel in the center of the patch.

Sliding window convents predicts one point at each iteration. Animation by Kisuk Lee.
Description of the convnet architecture of N4. Table from Ciresan et al. 2012.

We can write that architecture in tensorflow with the following code.

You can start training the network by running:

To track the progress of training in your browser, launch tensorboard:

Once we think the network has learned the task you can create affinites for the test set with:

To produce labels from the predicted affinities we will run watershed.
The repo includes an implementation of watershed in the julia programming language. Here is how to install julia.

You can now visualize the the results of your training by running:

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store