Super-Resolution and Object Detection: A Love Story- Epilogue

Jake Shermeyer & Adam Van Etten

The effects of super-resolving 30cm and 60cm imagery to 15cm in Brasília, Brazil. Imagery courtesy of DigitalGlobe.

A Brief Overview:

In our previous four posts on this topic [1, 2, 3, 4], and recently released paper in CVPR EarthVision, we showcased the results of our study on super-resolution and its effect on object detection performance. In order to quantify the effect of super-resolution, we first established object detection performance curves as a function of sensor resolution (Figure 1). Our results showed that object detection performance decreases by 22–27% when resolution degrades from 30 cm to 120 cm, and another 73–100% from 120 cm to 480 cm when looking across broad object classes.

Figure 1: Object detection performance as a function of sensor resolution for unenhanced native resolution imagery (SSD and YOLT) for five unique classes of objects.

We then applied super-resolution with two methods across these five resolutions. We used a standard computer vision baseline and neural network- Very Deep Super-Resolution (VDSR) and computationally inexpensive CPU only technique- Random Forest Super-Resolution (RFSR). We enhanced imagery 2x and 4x across each of these resolution ranges and then tested the performance tradeoffs. Our findings showed that super-resolution was most effective in the finest resolutions and became less effective as imagery becomes coarser. Overall the technique is a computationally inexpensive (~0.16 ms per 544x544 pixel image) and an effective pre-processing step for object detection applications (Figure 2).

Figure 2: The performance boost from 2x Super-Resolution for YOLT (change in mAP versus the blue line in Figure 1). Note that the performance boost is only valuable in the finest resolutions.

Some Fresh Findings:

We reran a few of our experiments at the finest resolutions to ensure that our object detection algorithms were observing the same extents on the ground. In our initial experiments we found that large aircraft were being clipped in half in the 15cm super-resolved imagery, and thus negatively skewing results. With this small tweak in design, our new experiments showed something intriguing that we hadn’t previously explored: Enhancing imagery from 30cm or 60cm to 15cm GSD provides a powerful boost in performance.

Figure 3: The benefit of enhancing 30 and 60cm imagery to 15cm.

In Figure 3 we have three subplots: On the leftmost plot we display object detection performance with 30cm (Green) and 60cm (Red) native resolution imagery. On the center and right plots we present the performance boost of enhancing imagery from 30 to 15cm (Teal) and 60 to 15cm (Orange) with VDSR and RFSR respectively.

The results of our study showcase several interesting bits of information:

A) When super-enhancing 60cm imagery to 15cm, object detection performance is improved by 11% vs. native 30cm imagery and by 20% vs. native 60cm imagery.

B) When super-enhancing 30cm imagery to 15cm, object detection performance is improved by 14% vs. the native 30cm imagery.

C) Once super-resolved to 15 cm, object detection performance for 30 and 60 cm imagery is equivalent (within errors)

D) Both a neural network (VDSR) and a simpler less computationally expensive method (RFSR) can provide similar enhancements to object detection performance when enhancing data from 60 or 30cm to 15cm.

Conclusions:

In this post, we showcase our final conclusions and quantified some of the effects of super-resolution on object detection performance in satellite imagery. For a deeper dive check out our paper recently published in CVPR EarthVision or our previous blogs in this series.