Annotation techniques for large-scale volume EM datasets
Generating annotations from volumetric electron microscopic (EM) image datasets is important for the work of many neuroscientists. As these EM datasets massively grow in size to many terabytes, smart tools as well as collaboration workflows are required to obtain high-quality annotations. The resulting annotations can be directly used for visualization or quantification purposes or indirectly, by training or evaluating a Machine Learning (ML) system.
In this blog post, you will learn about several techniques for dense and sparse annotations as well as collaboration workflows. If you are looking for a tool, try out webKnossos, which is specialized for collaborative annotation of large 3D image datasets.
This post is based on my talk at AILS institute. Watch it on Youtube.
Dense volume annotations
I’m sure many of you have seen regular brush tools for creating dense volume annotations. The user draws around the object of interest in every single section. This process is not fast, but very accurate, because the user is giving every section their full attention. By using dedicated hardware, such as a Wacom tablet or iPad with Apple Pencil, this process can be accelerated.
Some tools offer a feature for copying the annotation of an object to the next section or interpolating between multiple sections. This works well if the object does not change dramatically in contour from one section to the next. This can be much faster, but may require an additional correction pass for high-quality annotations.
Merging over-segmentations
Many ML-based segmentation pipelines work with over-segmentations that are later stitched together. For such scenarios, there are tools that make it easy to connect overly split segments to create new ground truth data. For example, you may have a model that has been trained on a different dataset and does not generalize perfectly to your new dataset. However, it may provide a decent enough over-segmentation that can be corrected and then used to fine-tune the model on your new data.
Interactive labelling with integrated ML
There are also interactive tools available that integrate a ML-system in the annotation workflow. For example, Ilastik implements a live preview of a prediction based on sparse annotations that a user creates. Only a few squiggles are required to generate the first preview. Because of the preview, the user can decide where to add more training data to help the ML model achieve better predictions. After some iterations, the trained model will perform much better. This works best for small enough datasets, where the user can overlook the whole data to figure out where the current model delivers suboptimal results. Although this is a great process for generating decent annotations, perfect ones will require a lot of iterations.
Let the computer pick areas
With active learning tools, the ML system will suggest areas where it needs more annotations. For that, the ML model will output not only a prediction, but also a confidence score. Where there is a low confidence score, the region will be sent back to user further annotations. This is similar to the interactive labelling, however, the ML system decides where to add more annotations instead of the user. This makes the approach scalable to very large datasets. Unfortunately, this is still an area of active research and there is no available tool that implements active learning for large volume EM datasets.
Sparse annotations
For many use cases, it is not necessary to acquire dense volume annotations. For example, visualizations, such as nuclei density plots, can be achieved with just sphere approximations of nuclei. Quantifications, such as counting or measuring, can be performed with just points or line segments. The benefit of sparse annotations, of course, is that they are much faster to obtain. For some use cases, it may be entirely unfeasible to obtain dense annotations. For example, a manual dense volume annotation of a whole neuron in an EM dataset with synaptic resolution is unfeasible. However, tracing a skeleton approximation is fairly cheap. As Boergens et al. 2017 quantified, skeleton tracings are up to 150x faster to generate than volume annotations.
webKnossos features a unique annotation mode for fast skeleton tracing called flight mode. The data is projected onto a sphere for seamless navigation. Combined with fast data loading, very fast and accurate tracing is possible. This is currently, the fastest manual skeleton tracing tool available.
Sparse annotations are also great for evaluating ML segmentations. By adding some points or tracing skeletons, you will have enough data in a short amount of time to assess your model. It will not provide information whether the model got the exact contours of the object correct, but it will provide information about missed or superfluous objects. Skeletonized neurons can provide a global view on the split and merger rates of an automated segmentation.
Collaboration and quality control
An effective way of accelerating annotation work is to split it up among multiple people. When you have a dataset (or part thereof) to annotate, your can either split it up by bounding boxes or by objects. With the bounding boxes, you don’t need to know anything about your dataset upfront. Just chunk it up and distribute among your annotators. We often use this approach for obtaining segmentation training data with randomly sampled boxes. If you already know about the objects in your dataset, you can split the work accordingly. For example, with a list of seeds in somata, you can ask your annotators to start skeletonization from there.
Manage annotation projects
When working with many annotators, a task management workflow can be very helpful. That is why we implemented task and project management features into webKnossos. First, you create tasks with detailed descriptions (ideally video tutorials). Second, the users request new tasks from their dashboard. The system in the background will match qualified users with the available tasks from the pool. Third, there are dashboard for monitoring the progress of the annotation work as well as tools for combining and downloading the results.
Quality control through redundancy
We find that the best way to guarantee high-quality annotations is through redundancy. Simply, give the same task to multiple people. This will deal with issues resulting from a lack of attention, misbehavior, and (to some extent) ambiguity in the data. However, redundancy comes with its own issues. Annotations from different users will differ slightly. Even annotations from the same user may show differences. To deal with that, there are consensus routines available. For volume annotations, voxel voting (with connected-component analysis) works well. Intersections or averaging may also work well for specific use cases. For skeletons, there is RESCOP that has been published by Helmstaedter et al. 2013. With these measures in place, it will be very efficient to obtain required annotations even for the largest datasets.
Try webKnossos for yourself
webKnossos is the result of our experience with large-scale annotation projects. It is great for deep learning projects to generate training and evaluation data as well as for visualization of prediction results. webKnossos also integrates a manual annotation service to scale your next annotation project. Create a free account today.
Image credits
- EM data, Mouse Cortex by Motta et al., Science, 2019
- Ilastik screenshot, by A. Klemm, BioImage Informatics Facility, Youtube, 2020
- Skeleton tracing, Rat Medial Entorhinal Cortex by Schmidt et al., Nature, 2017
- Volume annotations, Mouse Cortex by Berning et al., Neuron, 2015
- EM data, Olfactory Bulb by Carles Bosch, Francis Crick Institute
