Automated Neuron Reconstruction

At scalable minds we specialize in automated biomedical image reconstructions with our collaborators; especially reconstruction of neural tissue. The recent publication of the “Dense connectomic reconstruction in layer 4 of the somatosensory cortex“ paper by the Max-Planck-Institute for Brain Research is a great opportunity to showcase our machine learning reconstruction pipeline Voxelytics.

In the following blog post, I will talk more about the steps required for large scale 3D serial-blockface electron microscopy (SBEM) neuron reconstruction. Our goal is to create a fully automated neuron reconstruction and Connectome matrix that allows for meaningful biological analyses. The reconstruction is done using our machine-learning neuron reconstruction software Voxelytics.

If you are just interested in looking at the results, then check our segmentation in webKnossos and explore the synaptic connections in the interactive Connectome matrix.

A visualization of the raw electron microscopy data of the somatosensory cortex overlaid with our dense volume reconstruction. Each colored segment represents one neurite.
EM Data: MPI for Brain Research. Segmentation: scalable minds

The Dataset

All the raw EM image data, ground-truth annotations and segmentation results were published by the authors on the project website: https://l4dense2019.brain.mpg.de

The dataset is 194 gigabyte in size and covers 61.8 µm × 94.8 µm × 92.6 µm of the layer 4 of the primary somatosensory cortex (S1) of a mouse (P28). Each voxel spans 11.24x11.24x28nm³. You can view the whole dataset online with webKnossos.

The publication includes the full dense, volume reconstruction of all neurons and their connections to one another (Connectivity matrix/Connectome). More specifically it includes segmentation of all neurites, such as axons, dendrites, synapses and the connectivity graph between synapses.

A 3D visualization of the neuron reconstruction from the paper authors. The background shows the connectivity matrix, called connectome.
Image Credit: Science, MPI for Brain Research

The Process

Our neuron reconstruction software uses a multitude of convolutional neural networks (CNNs) and many individual pipeline steps to cleverly combine the individual predictions into highly accurate 3D segmentations.

An overview of the many pipeline steps required to generate a connectome. We feed the raw EM data through several machine learning models to extract various features. The tissue segmentation of each neurite is enriched with type information and connectivity to neighbors (synapses) to calculate a connectivity matrix/connectome.

As you can see, a connectome reconstruction is a complex undertaking requiring the training and fine-tuning of many machine learning models, smart combinations of intermediary results and a lot of engineering to efficiently distribute compute tasks on a scientific compute cluster or the cloud.

Overall compute time for the dataset was about a week when executed on a single workstation. That would be 0.5–1 days on a moderate CPU compute cluster. Additionally, some of the neural network models were trained up to a week on a single machine with a powerful Nvidia V100 GPU.

Neuron Segmentation, Agglomeration & Type Information

The hardest challenge in neuron reconstruction is the dense tissue segmentation of the raw EM data. Generally speaking this a three stage process:

1. Prediction: A machine-learning model classifies each voxel to be either part of a cell (intra-cellular) or background/cell boundary. We use a modified 3D U-Net [6] architecture and our predictions are based on voxel-wise affinity values between neighboring voxels instead of binary neurite/background prediction.
2. Segmentation: Based on that, a watershed algorithm generates an initial segmentation of connected voxels. This is typically an over-segmentation with many segments per neuron. This initial over-segmentation already yields fairly large segments, i.e. less segments that require further processing during the agglomeration step.
3. Agglomeration: All segments are combined into larger segments — ideally reconstructing the full neuron — using some form of connected component algorithm or another machine learning system.
   We use a hierarchical agglomeration approach along a region-adjacency graph (RAG) for agglomeration. This is inspired by Funke el al, 2017 [3]. This approach already leads to having longer neural path lengths reconstructed without any errors compared to the SegEM [2]+ ConnectEM [1,4] baseline provided by the original authors. We use further post processing steps such as making sure there is only one soma per neuron, the masking of blood vessels, and other heuristics for fixing splits.

A snapshot of our dense tissue segmentation after agglomeration. Each color and hatching combination denotes a single neuron or partial neurite out of the 100s of thousand neurons found in the dataset.

With these steps we are able to report an expected run length of 98.7µm for axons which are typically much harder to reconstruct automatically due to their small cross-sectional diameter. This is a more than 10x improvement compared to the automated reconstruction of the original authors (see table below). The resulting reconstructions are accurate enough to derive biological insights and already serve as the basis for neuroscience research for our collaborators. Therefore, we did not follow up with any manual proof-reading steps.

In the original publication, the authors invested an additional 4,000 hours of manual corrections to fix mistakes in the segmentation [1]. They used the high-speed “flight” annotation mode in webKnossos [5] to verify agglomeration candidates.

Our method shows a more than 10x improvement in expect run length for the difficult task of axons reconstruction using a fully automated approach. Additional manual human annotations is able to improve on this but comes at a premium in time and cost.

In contrast to the research team, we do not explicitly do any spine head detection/attachment. Our measurements indicate that we already detect and correctly attach about ⅔ of all spine heads during the dendrite reconstruction.

A 3D rendering of several reconstructed neurites. Note, the many spine heads that are already attached in the process.

Synapse Detection

In order to map connectivity between our reconstructed neurons, we need to find the interfaces where they touch and interact with one another. In other words we have to reconstruct the synapses.

Our approach for synapse classification combines CNN regressions with a boosted decision stumps classifier. We report an F1 score of 0.90 on the evaluation dataset, which is on par with the SynEM classifier [4] used in the original publication.

Synapse detection in itself is only half useful. In order to generate an accurate connectome, we are also interested in the directionality of each synapse. We derive the direction by using the type information (axon, dendrite) acquired in an earlier step. Further we classify the synapse types (spine-head, dendritic shaft, soma) — an essential information for scientific analysis and biological insights.

A snapshot of the synapse predictions. Our model does a voxel-wise prediction of the synapse interfaces.

The Connectome

With the neurite segmentation and synapse detection done, we have all the information to compute the connectivity matrix, the Connectome. In the image below, you can see all connections between the postsynaptic targets (dendrite shafts, spine heads, somata) and presynaptic axons.

Our connectome contains 452,960 synapses between 99,758 presynaptic axons and 159,545 postsynaptic dendrites. (Small/Large outliers were filtered for clarity in the image and interactive demo below resulting in only ~96,000 visible synapses.)

We reconstructed 453.000 synapse connections between neuron pairs and visualized them as a Connectome matrix (left side). Each Connectome entry can be viewed in the original data including the segmentation and 3D mesh visualization (right side). Interactively explore the connectome viewer yourself.

You can interactively explore the results in our connectome viewer on webKnossos at https://connectome.webknossos.org/scalable_minds/l4dense_motta_et_al_demo_v2/

Clicking on a Connectome entry will load the matching neurite segmentation and move you to the position in webKnossos. (For entries with more than one connection, you are taken to the first synapse connection.)

A sample of the reconstructed neurons. The dense nature of neural tissue leads to many interconnections (synapses) between neuron cells. Ultimately, the goal of the project is to extract and study these complex networks and derive biological and medical insights.

Discussion

Fully automated reconstruction methods for electron microscopy images have evolved greatly in recent years. The rapid evolution of machine learning methods such as convolutional neural networks have made it possible to analyse these large-scale datasets in a reasonable timeframe and without the need for thousands of hours of human reconstruction, a trend that enables the next generation of large-scale EM projects.

Our high-quality reconstructions are sufficient to address a large range of biological questions. For the case where the demand for accuracy is even higher, webKnossos can be used for further proofreading. View our reconstruction online on webKnossos.

We also offer our technology and knowledge as a service for neuron reconstruction of your data. We have successfully applied our methods on serial block-face EM (SBEM), multiSEM and ssTEM datasets. Check out Voxelytics on our website.
If you are interested in getting your EM data reconstructed with AI then contact us at hello@scalableminds.com

We would like to thank our research partners at the Max Planck Institute for Brain Research for making the data available and their collaboration in developing the automated methods in Voxelytics.

References

1. Dense connectomic reconstruction in layer 4 of the somatosensory cortex. Motta et al. Science, 2019
2. SegEM: Efficient Image Analysis for High-Resolution Connectomics. Berning et al. Neuron, 2015
3. Large Scale Image Segmentation with Structured Loss based Deep Learning for Connectome Reconstruction. Funke et al. archivX, 2017
4. SynEM, automated synapse detection for connectomics. Staffler et al. eLife, 2017
5. webKnossos: efficient online 3D data annotation for connectomics. Boergens et al. Nature, 2017
6. U-Net: Convolutional Networks for Biomedical Image Segmentation. Ronneberger et al. archiveX, 2015