SpaceNet 6: Announcing the Winners

Jake Shermeyer
Jun 4 · 6 min read
Image for post
Image for post
The SpaceNet 6 winner’s proposals (Green Outline) overlaid on the ground truth building footprints (Orange).

Preface: SpaceNet LLC is a nonprofit organization dedicated to accelerating open source, artificial intelligence applied research for geospatial applications, specifically foundational mapping (i.e., building footprint & road network detection). SpaceNet is run in collaboration by co-founder and managing partner, CosmiQ Works, co-founder and co-chair, Maxar Technologies, and our partners including Intel AI, Amazon Web Services (AWS), Capella Space, Topcoder, IEEE GRSS, the National Geospatial-Intelligence Agency and Planet.

The SpaceNet 6 challenge is officially in the books, and today we are proud to announce our winners! In the challenge, participants were asked to automatically extract building footprints with computer vision and artificial intelligence (AI) algorithms using two unique modalities of very-high resolution remote sensing data: synthetic aperture radar (SAR) from Capella Space and electro-optical satellite imagery from Maxar. Our area of interest for this challenge was centered over the largest port in Europe: Rotterdam, the Netherlands.

Image for post
Image for post
Image for post
Image for post
The formation of the SpaceNet 6 dataset: On August 4th, 22nd, and 23rd 2019, an aircraft mounted Capella Space sensor imaged the port area of Rotterdam. The same area was repeatedly captured in 204 successive passes from both north and south facing look-angles (35°). Then on August 31st, 2019 the Maxar WorldView 2 Sensor collected a full image of the same area at a near-nadir look angle (17°). On the left portion of this figure we can see the relative look angles of each of these collects. On the right, we can see a portion of the SAR data as it is collected on August 4th, overlaying the WorldView 2 Optical Image.

The SpaceNet 6 dataset is novel and the first permissively licensed dataset containing over 100 km² of very high-resolution half-meter SAR imagery. SAR sensors are unique as they can penetrate clouds and work in any illumination setting (day or night). As such, these sensors could be particularly valuable in a disaster-response scenario where cloud cover and adverse conditions often limit the value of traditional optical imagery. The SpaceNet 6 SAR imagery is collected via a test run of the Capella Space SAR sensor via an aerial platform. The same area of Rotterdam was imaged in 204 passes over 3 days, resulting in a dense stack of SAR data. Following these collects, just 8 days later, the Maxar WorldView 2 sensor passed overhead collecting a pristine image of Rotterdam. Together, along with our annotations of building footprints, these two remote sensing components form the basis of the SpaceNet 6 dataset and challenge.

Remember, that the training dataset contains both optical and SAR data, while the testing sets contain only SAR data. We structure the dataset in such a way to mimic real-world scenarios where historical optical data may be available, but concurrent optical collection with SAR is often not possible due to inconsistent orbits of the sensors, or cloud cover that will render the optical data unusable. If you’re unfamiliar with any other aspect of the SpaceNet 6 challenge, or if you want to read further, check out the other blogs in this series below, or read our paper, set to be released as part of the CVPR EarthVision proceedings in a mid-June 2020:

The dataset remains openly available for download via the AWS Open Data program.

Overall, the SpaceNet 6 challenge featured over 1,600 submissions and 411 registrants, making it our most competitive challenge to date. This post will mark the first in the series of posts as we begin to investigate what worked, what didn’t and what’s coming next. Without further ado: let’s jump right in.

The Winners

The SpaceNet 6 challenge crowns a new champion: zbigniewwojna. A summary of the winners’ results alongside the baseline model are in the table below:

Image for post
Image for post
Competitors’ scores in the SpaceNet 6: Multi-Sensor All Weather Mapping Challenge compared to the baseline model. The overall score represents the SpaceNet Metric (x 100) for the entire scoring set. We also report model precision (ratio of false predictions) and recall (ratio of missed ground truth polygons). The SatShipAI team was known as IoannisNasios on the TopCoder challenge leaderboard.

Our graduate winner is the __EVER__ team and our undergraduate winner goes to ikibardin. The margin of victory (2.4 points) by zbigniewwojna is actually the second largest in SpaceNet history, trailing only XD_XD’s win (5.0 points) from SpaceNet 2. XD_XD was the first in the challenge series to employ an ensemble of deep-learning models. This approach has since been the standard technique that has been used by our winners from SpaceNet 2 through 6. The winners were able to make significant improvements over the baseline and create their own unique solutions.

Image for post
Image for post
The SpaceNet 6 winner: zbigniewwojna proposals (Green Outline) overlaid on the ground truth building footprints (Orange). All ground truth is a modified version of the 3DBAG dataset. Models were most successful with structures ≥ 200m² with rectangular shapes. Note the offset of proposals relative to the ground-truth, a common challenge of off-nadir imagery.

Although the winning score is in the low 40’s, this is actually comparable to the results of the SpaceNet 4 challenge from very off-nadir look angles. The SAR data for SpaceNet 6 is also collected from an off-nadir perspective averaging* ~35°. These oblique perspectives are one of the more difficult aspects present in earth-observation and algorithms still have a long way to go to contend with them. Additionally, for this challenge, precision scores are actually much higher than recall scores. Overall, one out of every two proposed building footprints was a false positive and two out of every three buildings was missed by the winning algorithms. These numbers highlight the challenges of entering into a new modality and showcase that further research is required to fully exploit the SAR domain.

Model Architectures and Approaches

We summarize the winners algorithmic approaches in the following table:

Image for post
Image for post
The model architectures, ensemble and pre-training schemes, as well as training and inference time for each of the winning solutions. Note that the total contribution to the total NN’s ensembled is listed in parentheses in the Architectures column. Total training and inference times are calculated based upon an AWS p3.8xlarge instance with 4 NVIDIA Tesla V100 GPUs.

Takeaways:

  • EfficientNet Dominates: Each of the top 5 all used ensembles of neural networks, with four of the five relying on slight variants of the newly introduced EfficientNet. EfficientNet has achieved state-of-the-art performance on ImageNet while being markedly smaller and faster than other state-of-the-art convolutional neural networks.
Image for post
Image for post
The SpaceNet 6 winner: zbigniewwojna proposals (Green Outline) overlaid on the ground truth building footprints (Orange). All ground truth is a modified version of the 3DBAG dataset.

What’s coming next?

The SpaceNet team will be virtually attending CVPR EarthVision and will have two keynote slots: 11:40 PDT AM and PM on June 14th, 2020. In our keynote(s) we will be breaking down the challenge, the dataset, and the algorithms. Watch our social media (@Spacenet_AI), the EarthVision website, or the SpaceNet.AI website for live updates. Also coming up, we will have a series of blogs focused on this challenge as we begin to break down where these algorithms succeeded and what caused them to struggle. Finally, we will be open sourcing the top 5 algorithms on the SpaceNet Github repository in the coming weeks. Congratulations again to the winners and lookout of an upcoming announcement on SpaceNet 7!

*Note: This blog was updated from the initial version as SAR look angles were improperly cited as 55° rather than 35°.

The DownLinQ

Welcome to the official blog of CosmiQ Works, an IQT Lab…

Thanks to Adam Van Etten

Jake Shermeyer

Written by

Research Scientist at CosmiQ Works

The DownLinQ

Welcome to the official blog of CosmiQ Works, an IQT Lab dedicated to exploring the rapid advances delivered by artificial intelligence and geospatial startups, industry, academia, and the open source community

Jake Shermeyer

Written by

Research Scientist at CosmiQ Works

The DownLinQ

Welcome to the official blog of CosmiQ Works, an IQT Lab dedicated to exploring the rapid advances delivered by artificial intelligence and geospatial startups, industry, academia, and the open source community

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store