Super-Resolution and Object Detection: A Love Story- Part 2

Jake Shermeyer & Adam Van Etten

Jake Shermeyer
Nov 15, 2018 · 4 min read

In our previous post, we unveiled an introductory outline for some of our work in exploring the relationships between super-resolution (SR) and object detection algorithms in satellite imagery. As previously stated, we ultimately generate enhancement levels of 2x, 4x, and 8x over five distinct resolutions ranging from 30cm to 4.8m. Additionally, we produce a unique enhanced product: 15cm GSD super-resolved imagery. This post showcases more results, both qualitative and quantitative.

We chose the xView Dataset for the application of super-resolution techniques and quantification of object detection performance. The imagery consists of 1,415 sq. km of DigitalGlobe WorldView-3 pan-sharpened RGB imagery at 30cm GSD resolution spread across 56 distinct global locations. The labeled dataset for object detection contains 1-million object instances across 60 classes annotated with bounding boxes, including various types of buildings, vehicles, planes, trains, and boats. For our purposes, we ultimately discarded larger objects such as buildings and aggregated the dataset into 5 specific transportation classes including small vehicles, large vehicles, small aircraft, large aircraft, and boats.

All data were preprocessed consistently to simulate coarser resolution imagery and test the effects of our SR techniques on a range of resolutions. We attempt to simulate coarser resolution satellite imagery as accurately as possible by simulating the point-spread function (PSF) and using a more robust (inter-area) decimation algorithm. We intend our results to showcase what is reasonably accomplishable given coarser satellite imagery; rather than simply study what is possible given the ideal settings (no blurring & use of bicubic decimation) under which most new SR algorithms are introduced.

Our data were degraded from the native 30cm GSD using a variable Gaussian blur to simulate the PSF of the satellite depending upon our desired output resolution. A base Gaussian sigma of 1 was chosen and then multiplied by the scale of degradation. The more an image is degraded, the larger a Gaussian blur is initially applied. The inter-area decimation was then chosen to degrade the imagery from 30cm to resolutions of 60cm, 1.2m, 2.4m, and 4.8m. Each resolution was then trained using these images for levels of 2, 4, and 8x enhancements.

Image for post
Image for post
How does the world look at different resolutions? 30cm, 60cm, 1.2m, 2.4m, and 4.8m GSD resolutions featured.

As a reminder, all of our source code can be downloaded here:

Very Deep Super-Resolution For Geospatial (VDSR4Geo)

Random Forest Super-Resolution (RFSR)

Finally, we present some preliminary results and scores in terms of PSNR and SSIM, which are standard performance metrics for evaluating super-resolved outputs vs. ground truth imagery (Table 1). The next post will feature the final piece of the puzzle: the official results and findings on the relationships between object detection performance and super-resolution outputs.

Image for post
Image for post
Table 1. Quantitative evaluation of super-resolution performance (PSNR/SSIM scores for the luma component) reported for the xView validation dataset (281 WV3 Images). Although these are strong scores, pulling out the finest of features is necessary for stronger object detection performance.
Image for post
Image for post
This image depicts the native and super-resolved output in a tabular format. As resolution degrades, super-resolution is less effective at recovering higher-resolution details.

One of our primary findings from this work is that super-resolution is much more difficult in coarser resolution imagery. In these resolutions, mixed pixels become prevalent and small objects cannot be recovered. In our final blog(s) in this series, we will showcase the results of object detection performance of the super-resolved and native resolution imagery. Special thanks to Adam Van Etten, Dave Lindenbaum, Ryan Lewis, & Nick Weir for their contributions.

The DownLinQ

Welcome to the official blog of CosmiQ Works, an IQT Lab…

Thanks to Adam Van Etten

Jake Shermeyer

Written by

Research Scientist at CosmiQ Works

The DownLinQ

Welcome to the official blog of CosmiQ Works, an IQT Lab dedicated to exploring the rapid advances delivered by artificial intelligence and geospatial startups, industry, academia, and the open source community

Jake Shermeyer

Written by

Research Scientist at CosmiQ Works

The DownLinQ

Welcome to the official blog of CosmiQ Works, an IQT Lab dedicated to exploring the rapid advances delivered by artificial intelligence and geospatial startups, industry, academia, and the open source community

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store