Artificial Colorization of Grayscale Satellite Imagery via GANs: Part 2
Introduction
In Part 1 of this blog series, we outlined a method to artificially color grayscale satellite images using a generative adversarial network (GAN).
Multi-task Network Cascades (MNC) is a popular object segmentation algorithm that won first place in the MS COCO 2015 competition. In a previous blog post, Object Segmentation on SpaceNet via Multi-task Network Cascades (MNC), we discussed the results of applying MNC to building footprints in 3-band color satellite imagery. In this blog post we show that MNC is better at extracting building footprints in artificially colored satellite images than in their underlying grayscale images. Thus, colorization may be interpreted as a preprocessing step to grayscale satellite images that adds value for building footprint detection with MNC.
Performance as a Function of Imaging Bands
In a previous blog post, we reengineered MNC and YOLT to train on all three SpaceNet image types: grayscale panchromatic imagery, pansharpened 3-band RGB color imagery, and pansharpened 8-band multispectral imagery. This resulted in 12 trained models.
The F1 scores of these 12 trained models (evaluated on test SpaceNet images) demonstrate that MNC performs better on pansharpened 3-band RGB color imagery than on 1-band grayscale imagery. The following chart summarizes these findings for MNC.
The small black bars in this chart are error bars for the F1 scores computed via bootstrapping. A more detailed description of these findings are described in Panchromatic to Multispectral: Object Detection Performance as a Function of Imaging Bands.
MNC Results on Artificially Colored Images
The experiment we perform is the following. For each city, we possess an MNC model trained on grayscale images and a second model trained on real 3-band RGB color images. Using the Colorization GAN introduced in Part 1 of this series, we produce inferred (or fake) color images from the grayscale images using the generative network.
We compare the results of the following three combinations of trained MNC models and test data sets:
- MNC trained on grayscale images and tested on grayscale images.
- MNC trained on real color images and tested on artificially colored images.
- MNC trained on real color images and tested on real colored images.
The following bar chart and table summarize the results of these experiments:
In summary, we find that MNC performance improves significantly from grayscale images to artificially colored images. However, MNC still performs better on the original color satellite images than on artificially colored images. That is, we cannot fully recover all of the building footprint detections by colorizing grayscale images.
Visualizations of MNC
We now include visualizations of MNC building footprint detections on grayscale, fake color images, and real color images. Each row of images represents a single 200 m x 200 m area over a single city: Las Vegas, Paris, Shanghai, and Khartoum.
In the pictures below, a white outline represents a true positive for MNC, and a yellow outline represents a false positive. A more detailed description of the SpaceNet metric can be found here. We choose not to outline false negatives in these images for visual simplicity.
Conclusions
Given a large corpus of grayscale satellite images over a city (say in storage) and a small amount of color satellite images over that same city (we used one 2500 m x 2500 m color image to train our GAN), artificially coloring these grayscale images could significantly boost the performance of building footprint detection with MNC.
Analogously, in a bandwidth limited environment, one could transmit a small amount of color imagery and transmit the rest as grayscale imagery, relying on the GAN to artificially color the grayscale imagery.
