Sentinel-2 Deep Resolution 3.0
S2DR3: Effective 12-Band 10x Single Image Super-Resolution for Sentinel-2
The article documents the performance of S2DR3 — a major update of our 12-band 10x Single-Image Super-Resolution model that upscales all 12 spectral bands of a single Sentinel-2 L2A (or L1C) scene from the original 10, 20 and 60 m/px spatial resolution to the target 1 m/px.
S2DR3 model is specifically optimised to preserve subtle spectral variations of soil and vegetation across all 12 multi-spectral bands of Sentinel-2 L2A and is capable to accurately reconstruct objects and textures with individual spatial features down to 3m.
The S2DR2 model has been released in late 2022 and found multiple useful applications, where the main demand has been driven by the evaluation of real-time data for precision farming including the detection of high-precision field boundaries provided by our partner DigiFarm. The peak demand for S2DR2 data has been achieved in June 2023 where 12 million sq. km of Sentinel-2 imaging data has been processed in a single month.
In my previous S2DR2 post, I have reflected on the substantial level of scepticism towards ANN-based super-resolution techniques that I have encountered from the Earth Observation community. I believe, such attitudes are slowly changing as the AI-based applications and practical solutions, such as ChatGPT become more common place. I also think that a fare dose of healthy scepticism is justified in the presence of purely generative methods such as DALL-E and Midjourney.
The objective of this post therefor is to continue to offer evidence and validation in order to establish S2DR3 as a viable source of high-revisit rate, radiometrically calibrated, high-resolution satellite imaging data that is suitable for practical analytical applications.
A number of similar solutions have emerged in recent months suggesting the importance of the subject to the Earth Observation community and amount of effort that is being applied to the problem by both academic and commercial players. Ultimately, the utility of such methods is confirmed by the emergence of practical applications such as the aforementioned high-precision delineation of agricultural fields by DigiFarm illustrated in Fig. 2. To the best of my knowledge, DigiFarm provides by far the most accurate and up-to-date field boundaries available today.
In our new version S2DR3 we have focused on the spectral fidelity of the super-resolved data. The detailed analysis of the attained performance, both spectral and spatial is presented in the next section, however I would like to argue that the high spectral fidelity of S2DR3 combined with the superb quality of the source Sentinel-2 L2A data will make S2DR3 an ideal source of remote sensing data for Environmental Monitoring, as well as Measurement, Reporting and Verification (MRV) applications for the burgeoning sector of carbon credits.
S2DR3 was specifically optimised to preserve subtle variation in spectral characteristics of vegetation across all 12 bands of Sentinel-2 L2A multi-spectral imaging data. Consequently, out model is capable to generate valid results for any spectral analysis pertaining to soil and vegetation at 10x the resolution of the original Sentinel-2 data.
Network Architecture: The underlying ANN architecture of our method is not dissimilar to other SISR state-of-the-art methods. To the best of my knowledge, many such solutions apply invoke a fully convolutional network, such as the excellent RRDBNet proposed in ESRGAN or any of its variants.
It should be said however that in my experience four key factors have impact on the attainable performance of the model in the following order of decreasing importances
- Training data
- Loss function
- Hyperparameters
- Network architecture
S2DR3 Google Colab notebook available for testing and validation.
Qualitative Performance and Known Limitations
Spectral Fidelity: The development of the new version of our Sentinel-2 super-resolution model was largely motivated by the objective of further improving the spectral accuracy of the super-resolved data. As was described in our previous post, this is particularly challenging because the corresponding high-resolution multi-spectral ground truth imaging data does not exist and had to be artificially synthesised for the purposes of model training. Likewise, the accuracy of the end result is difficult to evaluate exactly due to the lack of proper ground truth. Consequently, our assessment is based on comparing spectral characteristics of the downsampled output to the original input.
Our rational is that the model is tasked to reconstruct morphologically and semantically coherent spatial details while preserving the spectral characteristics of the source data. Following our comprehensive assessment of the results we concluded that S2DR3 achieves an outstanding spectral fidelity, which make it suitable for wide range of important practical analytical applications, as further demonstrated by quantitative results presented in the following section. It should be said however that some deviations from the source spectral characteristics have been observed particularly in datasets with dense and unusual objects that were not well represented in training datasets.
Semantic Integrity and Hallucination: Throughout our extensive performance evaluation that included randomised sampling of global Sentinel-2 data we did not encounter any obvious instances of “hallucination”, namely generation of objects, such as buildings or trees that are not rooted in ground truth. Nevertheless, the model can often fail to reconstruct small objects, typically < 5m across, particularly in cases where such objects have low spectral signature against the background, or in other words, their colour is similar to the colour of the background.
Geometric Fidelity, Artefacts and Distortion: Small scale distortions of objects are common and particularly prominent in urban environments comprised by complex, high density patterns of regular geometric shapes characteristic to man-made objects. In the context of agricultural fields, the model demonstrates impressive capability to reconstruct regular periodic patterns even where the scale of individual structural features is under 3m (< 0.3 of a single Sentinel-2 pixel) illustrated in fig. 4. This remarkable capability has, of course, it’s own limits and the accuracy of reconstructed textures may vary depending on the scale of the underlying features and their respective spectral characteristics.
We are cordially inviting requests for independent evaluation and validation of the described results by the qualified experts and enthusiasts of the Earth Observation community. Please contact ya at gamma dot earth for more information.
Quantitative Performance Evaluation
We evaluate two separate aspects of the achievable performance. Firstly, the accuracy of the spatial reconstruction is evaluated across the RGB bands using high-resolution ground truth, and RMSE, PSNR and SSIM metrics. Secondly, the preservation of spectral characteristics across the 12 spectral bands of Sentinel-2 is evaluated using the R² score between the original and upscaled pixels. The main purpose of this step is to ensure that the model does not introduce spectral distortion or biases into the original spectral data.
We hence force present a comprehensive collection of examples generated across a broad range of locations, seasons and types of terrain that demonstrate and document the achievable performance, as well as the expected limitations of the model. All testing samples were selected to be distinctly different from the data that has been used during training. More specifically, the samples were taken from Sentinel-2 scenes that were not used in training.
Each example contains the following:
- Sentinel-2 True Color Image (TCI:B04,B03,B02) 40x40px 10m/px(top, left);
- S2DR3 TCI 400x400px 1m/px (top, centre);
- Ground Truth RGB 1m/px (top, right © Google Earth);
- Sentinel-2 Infra-Red Pseudo-Color Image (IRPCI:B12,B08,B05) 40x40px 10m/px(bottom, left);
- S2DR3 IRPCI 400x400px 1m/px (bottom, centre);
- Scatter plot of pixel values across all 12 bands, Sentinel-2 L2A versus S2DR3. Different colors in the scatter plot represent different spectral bands. Furthermore, the plot contains accuracy evaluation values, including metrics RMSE, PSNR, SSIM and R2
An extended collection of additional S2DR3 examples can be found here.
