RarePlanes - An Introduction
This past year CosmiQ decided to create a new dataset for object detection and instance segmentation of aircraft and their attributes in satellite imagery that we ultimately entitled “RarePlanes”. Throughout the dataset creation process we quickly learned that hand annotation of features in satellite imagery is difficult, expensive, time consuming, and potentially a mind-numbingly tedious task. After spending hours and hours labeling aircraft, correcting attributes, and listening to Radiohead’s most depressing songs well above the OSHA recommended decibel levels, I began to question my own existence. Why would I, like many other scientists and engineers the world over, subject myself to such dull and uninteresting work?
If you follow our work and this blog, the answer is probably unsurprising: to train data hungry computer vision algorithms! Supervised deep learning feeds off of the combination of human annotated labels and imagery, and typically the more well labeled data, the better the performance (at least for awhile). But what if there was a work-around to this laborious workflow? What if we could rely on an alternative method to generate labeled datasets? Thankfully there is an intriguing option that’s been recently introduced proposing the use of gaming engines to generate “synthetic” data. Synthetically generated datasets have been built for dozens of tasks, such as: aiding self driving cars, indoor 3D navigation, optical flow, and even robotic chefs.
Research has shown that such synthetic datasets can reduce the amount of real training data needed, and potentially improve performance for certain tasks. But geo is hard (Space Club Rule #36), and at CosmiQ, we don’t believe the hype until we test it ourselves. Consequently, we set out with the ambitious goal to build our own dataset to test out where synthetic data works, where it breaks, and provide data and methods to aid the development of a whole new set of vision tasks for the community to tackle.
Today we introduce RarePlanes, a machine learning dataset and research study that examines the value of synthetic data to aid computer vision algorithms in their ability to automatically detect aircraft and their attributes in satellite imagery. CosmiQ curated a dataset of ~600 WorldView-3 satellite images spanning over 200 locations in 31 countries. It includes ~30,000 manually annotated aircraft and 9 fine grain attributes including: aircraft length, wingspan, wing-shape, wing-position, propulsion, number of engines, number of vertical-stabilizers, if it has canards, and aircraft role. The accompanying synthetic dataset is generated via IQT portfolio company AI.Reverie’s simulation platform and features over 46,000 simulated satellite images with ~300,000 airplane annotations.
The experiments in RarePlanes will address three key areas:
- The performance tradeoffs of computer vision algorithms for detection and classification of aircraft role (e.g. light civil) /specific model (e.g. Cessna 172) using blends of synthetic and real training data.
- The performance tradeoffs of computer vision algorithms for identification of rare aircraft that are infrequently observed in satellite imagery using blends of synthetic and real training data.
- The value of weakly supervised annotations for detection of unique aircraft attributes.
These experiments should help to inform the best practices for using synthetic data in an overhead perspective, and be transferrable to other challenging detection tasks beyond aircraft.
I should note that just like SpaceNet a portion of this dataset will be open-sourced! Stay tuned for details on this and a blog series going deep on the dataset, the features, our experiments, and conclusions that should hopefully inspire some future research.
