PyTorch
Published in

PyTorch

Multi-target in Albumentations

Many images, many masks, bounding boxes, and key points. How to transform them in sync?

Image source: https://www.newsbreak.com/news/1480496874024/elon-musk-revealed-his-favorite-film-of-2019-was-parasite

I am one of the authors of the image augmentation library Albumentations.

Image augmentations is an interpretable regularization technique. You transform the existing data to generate a new one. (For more details check the great post about different augmentations libraries by Neptune: Data Augmentations in Python)

https://albumentations.ai/docs/introduction/image_augmentation/

You can use the library with PyTorch, Keras, Tensorflow, or any other framework that can treat an image as a numpy array.

Albumentations work the best with the standard tasks of classification, segmentation, object, and keypoint detection. But there are situations when your samples consist of a set of different objects.

Multi-target functionality specifically designed for this situation.

Possible use cases:

  • Siamese networks
  • Sequential frames in the video.
  • Image2image.
  • Multilabel segmentation.
  • Instance segmentation.
  • Panoptic segmentation.

A few buzzwords :)

For the past three years, we worked on functionality and optimized for the performance.

Now we focus on documentation and tutorials.

At least once a week users ask to add support for multiple masks.

We already have it for more than a year. :)

This article will share examples of how to work with multiple targets with albumentations.

Scenario 1: One image, one mask

The input image and mask.

The most common use case is image segmentation. You have an image and a mask. You want to perform a spatial transform of both image and mask, and it should be the same set of transforms. Albumentations takes care of this requirement.

In the following code, we apply HorizontalFlip and ShiftScaleRotate.

Scenario 2: One image and several masks

Input: one image, two masks

For some tasks, you may have a few labels corresponding to the same pixel.

Let’s apply HorizontalFlip, GridDistortion, RandomCrop.

Scenario 3: several images, masks, key points, and bounding boxes

You may apply spatial transforms to multiple targets.

In this example, we have two images, two masks, two bounding boxes, and two sets of keypoints.

Let’s apply the sequence of HorizontalFlip and ShiftScaleRotate

Q: Can we work with more than two images?
A: You can use as many as you want.

Q: Should the number of image, mask, bounding box, and keypoint targets be the same?
A: You can have N images, M masks, K key points, and B bounding boxes. N, M, K, and B could be different.

Q: Are there situations where multi-target will break?
A: In general, you can use the multi-target functionality to a set of images of different sizes. Some transform depends on the inputs. For example, you cannot perform a crop that is larger than the image. Another example is MaskDropout that depends on the input mask may. How will it behave when we have a set of masks is unclear. We will test these corner cases. But they are pretty rare in practice.

Q: How many transforms could be combined together?
A: You can combine the transforms into a complex pipeline in a number of ways.

We have more than 30 spatial transforms. All of them support images and masks, most of them support bounding boxes and key points.

https://albumentations.ai/docs/getting_started/transforms_and_targets/#spatial-level-transforms

That could be combined with 40+ transforms that modify pixel values of the image. Example: RandomBrightnessContrast, Blur, or something more exotic like RandomRain.

More documentation

Conclusion

Working on the open-source project is challenging, but very exciting. I would like to thank the core team:

and all the contributors that helped to build the library and get it to its current state.

--

--

--

An open source machine learning framework that accelerates the path from research prototyping to production deployment

Recommended from Medium

The 4th Knowledge Revolution: AI-Powered Chatbots

Fintech for Negotiating Debts: Five Options to Make Debt Collection Easier

[IJCAI 2018] Virtual-to-Real: Learning to Control in Visual Semantic Segmentation

AI Must Have a Cruciform Ethic

10 Sci-Fi and Fantasy Writing Prompts to Get Your Creativity Flowing

Weekly Digest: Cortana Revamp, Improv-based Chatbot, and ‘Hands-Free’ Mobile Alexa

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Vladimir Iglovikov

Vladimir Iglovikov

Ternaus.blog

More from Medium

A seamless way of tracking ML experiments: Weights & Biases

Fake News Classification using Passive Aggressive Classifier and LIME

Building a Playstation Network Recommendation System from Scratch

AI to the Rescue of the Agent’s Energy Levels