2nd SpaceNet Competition Winners Code Release
In June, we concluded the second SpaceNet competition, a satellite imagery object detection competition hosted by CosmiQ Works, DigitalGlobe, and NVIDIA. This competition challenged contestants to submit algorithms that automatically detect building footprints in satellite images, continuing the challenge originally posed in the first SpaceNet competition.
As explained in a prior post, our second competition used a new dataset consisting of satellite imagery from WorldView-3 and labeled building footprints over four cities: Las Vegas, Paris, Shanghai and Khartoum. Details about the dataset can be found in that post, including how it differs from that used in the first competition.
The source code of the top three winners is available on the SpaceNet GitHub repository. Each solution comes with instructions to build a Docker container, which can in turn be used to train models and perform inference on new satellite data.
SpaceNet Round 2 Results
The winning solution was developed by TopCoder XD_XD with a final footprint evaluation metric score of .693 averaged across the four cities. The top three contestants all performed well on Las Vegas, but struggled on Shanghai and Khartoum (see individual F1 scores below):
In this competition, a building footprint detection is based on an Intersection over Union (IoU) of 0.5 or greater. Figure 2 shows the distribution of IoU scores for the building footprints on the test set of satellite image chips. XD_XD’s solution outperforms other competition submissions both on overall F1 score and on average individual IoU scores for each building footprint. While average individual IoU scores (exceeding 0.5) do not directly impact the overall F1 score, they do indicate that XD_XD’s solution produces precise building outlines.
The Winning Implementation: XD_XD
XD_XD’s winning solution is an ensemble of three U-Net neural networks. He uses geospatial information embedded in training data to integrate additional OpenStreetMap data into his model. You can read more about XD_XD’s solution here. The full write up and code can be found here.
XD_XD’s solution is visually compelling. In Las Vegas, his solution excels at outlining suburban homes. In Khartoum, the solution performs well on stand-alone apartment complexes. However, XD_XD’s solution has trouble with small buildings, with building footprints that are close together, and with large buildings in Khartoum.
Second Place Implementation: Wleite
Wleite achieved second place using his code base from the first competition while providing a more methodical evaluation of possible improvements. His implementation is a custom java application that does not leverage deep learning frameworks. It consists of a four-part process:
1. He applies edge detection to each band of the multiband image.
2. He uses random forests to create two binary classifiers. The first classifier determines whether a pixel belongs to a border and the second is whether a pixel is inside or outside a building.
3. The brute force polygon matching is performed based on the edge detection and pixel classification.
4. Each polygon candidate is evaluated for its probability of having an IoU over 0.5, and polygons below a certain city-dependent cut-off are discarded.
You can read more about his solution here.
Third Place Implementation: Nofto
Nofto achieved third place in the competition using Wleite’s code from the first SpaceNet competition. Nofto modified the code to use the multi-spectral pan-sharpened imagery and modified the random forest implementation. The full write up and code can be found here.
Conclusions
The results from 2nd SpaceNet competition show that performance is regionally specific. While the competition submissions excel in locating building footprints in Las Vegas, the submissions struggle in Shanghai and Khartoum. We encourage readers to explore the winning solutions developed by XD_XD, Wleite, and Nofto. The dataset will continue to be available on SpaceNet’s AWS page and the solution code base will remain available on the SpaceNet Github repository.
Going forward, we are planning new SpaceNet competitions beyond automated building footprint extraction. We will launch a road network competition this fall for the 3rd SpaceNet competition, and we will provide details regarding this competition soon.
