The SpaceNet 7 Multi-Temporal Urban Development Challenge: Announcing the Winners
Preface: SpaceNet LLC is a nonprofit organization dedicated to accelerating open source, artificial intelligence applied research for geospatial applications, specifically foundational mapping (i.e., building footprint & road network detection). SpaceNet is run in collaboration by co-founder and managing partner CosmiQ Works, co-founder and co-chair Maxar Technologies, and our partners including Amazon Web Services (AWS), Capella Space, Topcoder, IEEE GRSS, the National Geospatial-Intelligence Agency and Planet.
The 7th iteration of the SpaceNet Challenge Series is officially in the books, and our top 5 winners have been determined. The Multi-temporal Urban development Challenge staked out an ambitious goal: identify and track building footprint change over a deep temporal stack. We are happy to report that this challenging task was met with a high degree of ingenuity by the top competitors. Our public baseline model achieved a score of 17 (out of 100) on the final test set, yet the winning submission more than doubled this score, with a score of 41. In the following sections we detail the winners of the challenge, and provide a prelude of some of the insights gleaned from this challenge (further analysis will have to wait for subsequent blogs).
This is the first SpaceNet challenge to explicitly include the time dimension, and accordingly used a new medium resolution satellite imagery dataset from Planet that encompasses over 100 locations around the globe and 24 consecutive monthly observations that cover multiple seasons and significant construction in most locations. See here for more details about the dataset.
As noted in previous blogs [e.g. 1] the task of identifying buildings and tracking unique building identifiers (i.e. addresses) over time has numerous humanitarian, urban planning, and disaster response applications. We constructed a new metric, SCOT, to measure performance in tracking buildings, and released a baseline algorithm demonstrating that the ambitious goal of SpaceNet7 was indeed achievable.
Over the course of the previous six SpaceNet challenges, we have observed significant improvements in algorithmic performance. Much of these improvements stemmed from careful curation of training datasets and adoption of state-of-the-art neural network architectures. Less effort has been spent on algorithm design; for example in SpaceNet 5 all competitors used the CRESI algorithmic framework, but swapped out segmentation models to achieve small performance gains. SpaceNet 7, however, yielded significant variation in approaches, and creative approaches to post-processing that did not simply rely upon the tried-and-true (though uninteresting) fallback of adding yet another model to an ensemble. In fact, the winning model did not use an ensemble of neural network architectures at all, and managed an impressive score with only a single, rapid model. Table 1 details the top-5 prize winning competitors of the 300+ participants in SpaceNet 7.
We see from Table 1 that ensembles of models are not a panacea, and in fact post-processing techniques have a far greater impact on performance than the individual architecture chosen. We will explore the novel post-processing techniques used in a later blog. For now, we note that the winning algorithm is a clear leader when it comes to the combination of performance and speed, as illustrated in Figure 1.
In addition to the top-5 overall winners, the top undergraduate and top graduate student teams also receieved prizes. The top undergraduate prize goes to team SJTJ, ranked #13, with members: LAOS_Y, maxLeung, SKFE, Tennant. The top graduate prize goes to team CIVASN, ranked #10, with members: rbync, imadtoubal, noor_mus99, wyyqwqq.
In Figures 2 and 3 we show predictions from the winning algorithm.
3. Preliminary Analysis of Approaches
As noted above, post-processing techniques are really where the winning submissions differentiated themselves (and will be covered in depth later), but there are a few trends in the initial deep learning segmentation approach worth noting.
3A. Upsampling Improved Performance
The moderate resolution of imagery poses a significant challenge when extracting small footprints, so multiple competitors upsampled the imagery 3–4x and noted improved performance (see Figure 4).
3B. 3-channel Training Masks
The small pixel sizes of many buildings results in very dense clustering in some locations, complicating the process of footprint extraction. Accordingly, multiple competitors found utility in a multi-channel training mask for deep learning segmentation that incorporates building spacing, see Figure 5.
3C. Ensembles Remain the Norm
While the winning algorithm eschewed multi-model ensembles (to great speed benefits), the remainder of the top-4 competitors used an ensemble of segmentation models. A typical approach is illustrated below.
The seventh iteration of the SpaceNet Challenge is in the books, with rookie competitor lxastro0 notching the win. While beyond the scope of this blog to dive in too deeply, we look forward to reviewing the creative algorithms used to solve the difficult problem of tracking building footprints in medium resolution imagery. At the end of the day, the winning score of SCOT = 0.41 is extremely impressive, and subsequent blogs will dive further into algorithmic details, failure modes, and success stories.