The importance of manual annotations for EM-based neuroscience

Researchers in nano-scale neuroscience, such as in Connectomics, are dealing with huge amounts of data delivered by the electronic microscopes, which need to be reconstructed in order to extract meaningful scientific insights.

In some cases, sparse neuron annotations might be enough. Most of the time, however, you will need to generate high quality evaluation data to train a Machine Learning (ML) model. Either way, a reconstruction of the brain’s cells starts with high quality manual annotations.

When starting from scratch, this annotation process can be time-consuming, expensive, and difficult. Finding qualified annotators is a real challenge. Thanks to our experience in many annotation projects and our extended knowledge in nano-scale neuroscience, we are able to offer high-quality manual annotation services. Our services will save you time, money, and head aches.

4 human annotators segmenting brain cells on EM data thanks to webKnossos’ volume annotation tools.

In this blog post, I will explain why manual annotations are essential for neuroscience, describe use cases, provide some practical details on how our annotation services work and lastly, showcase some projects we performed.

Use cases of manual annotation in neuroscience

Generate training data for segmentations

EM data manually annotated by our team in order to generate training data for a Machine Learning model. Raw EM data by Motta et al.

To reconstruct biological objects from large-scale EM images, you will have to use Machine Learning. As described above, this starts with generating evaluation and training data. Our annotators can create dense segmentations for bounding boxes within your dataset, which can then be used to train your Machine Learning systems for automatic segmentation. Features of interest can be any biological object from dense neurites, nuclei, blood vessels, other organelles to larger features such as brain regions, lesions or entire organs. If you need help with automated segmentations, check out our machine learning reconstruction pipeline Voxelytics.

Skeletonize neurons

A typical use case for manual annotations is to skeletonize neurons. The annotators will generate skeleton traces from neurons in your data. These sparse annotations can later on be used directly for scientific discovery or in conjunction with automated analysis, e.g. for evaluation purposes or as training data.

Skeletonized neurons traced by our team of annotators. The resulting sparse annotations can serve directly as scientific discovery or later as training data for ML.

Object detection

By clicking on every nucleus, every synapse, every mitochondria and so on, trained annotators generate object detections. Again, this annotated data can serve as is or be combined with automated analysis.

Here, an annotator is manually placing nodes on each nucleus to detect objects. Raw EM data by Motta et al.

Type annotations

In some situations, you might already have an existing segmentation and need classification. In this case, the annotators will give each segment a type (for instance an axon, dendrite, nucleus, and so on).

Proof-read automatic segmentations

Annotator correcting splits errors manually thanks to the merger mode in webKnossos. Raw EM data by Motta et al.

In the context of an automated segmentation, manual annotations are not only useful to generate training data, but can also help to fix errors. Indeed, state-of-the-art automatic segmentation methods generate good segmentations, but still produce split and merge errors. These can be efficiently corrected by a good team of annotators through webKnossos’ advanced proof-reading tools.

Ensuring quality

As the basis of automated analysis or scientific discovery, the quality of the annotations is most important. With the experience of many annotation projects, we have implemented several processes to ensure a high quality of data.

Annotator selection and training

We select our annotators based on previous experience with biomedical images. We constantly improve our training materials and provide detailed feedback on sample tasks. Additionally, we continuously monitor the work of each annotator.

Redundancy

Depending on the performed task and quality requirements, we perform the task redundantly by separate annotators. This can range between 2- and 5-fold redundancy. With our consensus technology, we will provide unified results to you. This works both for volume annotations as well as for skeleton tracings.

To ensure quality, our annotators are working redundantly on the same task. Raw EM data by Bosch (Francis Crick Institute)

Annotator and reviewer roles

In this mode, one person will perform the annotation job and another person will review and correct the work. Reviewers will be specifically qualified annotators, who have proven their reliability in previous projects.

In order to avoid mistakes and ensure quality, our annotation services include reviewers: one annotators will review and correct the work performed by another.

How our service works

With your data on webKnossos, we will schedule an intro call to discuss your project’s needs. Based on that, we will propose a project structure and estimate. We will then get started with our team of annotators.

Since all the work is happening in webKnossos, you will be able to review intermediate results and track the overall progress of the project from start to finish.

When the project is complete, the results will be available in webKnossos from where you can continue working with them or download in your desired format.

Examples from Connectomics

Skeletonizing apical dendrites in EM data

In this project, our annotators generated skeletonizations of apical dendrites in the olfactory bulb of a mouse. The raw data was serial-blockface electron microscopy (SBEM) images. The annotators were presented with pre-seeded somata and their task was to find the apical dendrite and accurately generate skeletons from the soma in the mitral cell layer (MCL) to their ending in the glomular layer (GL).

Apical dendrites skeletonized by our annotation team. Raw EM data by Bosch (Francis Crick Institute).

In total, 301 apical dendrites were annotated. The project was delivered within a month, and we implemented a 5-fold redundancy for quality assurance. The overall budget was 3,000 EUR.

Dense neuron segmentation

Brain cells manually segmented by our team of annotators, resulting in a dense neuron segmentation.

For this project, a dense reconstruction of neural tissue was needed. Our annotators generated training data thanks to webKnossos’ volume annotation tools within several bounding boxes of the raw dataset. Special attention was given to annotate all branches in the neurons and carefully segment even the smallest and thinnest processes.

The project was delivered in 2 months and the budget was 6,000 EUR.

Type annotations on brain tissue segmentation

For this project, a dense segmentation was already at hand. The annotators had to define the type of the segments. Using webKnossos to visualise and annotate the data, they defined for each segment whether it was an axon, dendrite, astrocyte, soma, and so on.

To ensure quality, we implemented a 2-fold redundancy. In total, 2,000 segments were annotated for a budget of 2.000 EUR. The project was completed in 2 weeks.

With a dense segmentation already available, our annotators had to define the type of each segment: astrocyte (blue), axon (pink), dendrite (green), spinehead (yellow), etc. Raw EM data by Motta et al.

Data annotation, beyond Connectomics

With our experience and knowledge in generating annotations for nano-scale neuroscience, we are qualified for further scientific data annotations. For instance, we worked on projects in material science and meteorology.

Annotating microscopic crack patterns on steel

Skeleton annotations on steel’s microscopic cracks by our annotation team.

The goal of this project was to annotate the steel’s surface cracking at a microscopic scale. Our team had to annotate several datasets obtained by scanning electron microscopy (SEM). These annotations allowed the researchers to study the density, morphological and topological features of the crack patterns.

The result were skeletons of the inter-connected crack patterns. The annotations were delivered within 2 weeks.

Nuclei segmentation for pathology: plasmacytoma

Our annotators were asked to annotate plasma cells on micrographs stained with haematoxylin and eosin. Thanks to webKnossos’ volume annotation tools, they outlined the nuclei and delivered a segmentation with number and position of the plasma cells, necessary in the context of the diagnosis of plasmacytoma.

Volume annotations by our team on micrographs of plasma cells.

We are always glad to discover new fields and use cases. Our annotators are flexible and are eager to learn about all new kinds of data.

Thanks to our annotation services, you will get quality-controlled annotations from our expert annotators specialised in neuroscience without having to manage the project yourself. Get in touch with us to discuss the specifics of your annotation needs. We are looking forward to hearing from you!