Reproducing the result of paper Burst Denoising of Dark Images

Photo from pixabay.com

Image Processing faces various challenges such as object detection, object classification, de-blurring, etc. Processing image captured under low-light condition is a similar challenging problem. In this blog, we present our implementation based on the research paper Burst denoising of dark images (https://paperswithcode.com/paper/burst-denoising-of-dark-images) and show some of our results as well as the limitations.

Comparison between noisy-raw image(left), SID[2] result(second from left), result from this paper(third from left), groundtruth(right) [1]

Introduction

In the past, traditional pipeline was used to denoise raw image and enhance image quality (Figure 1). However, with the emergence of deep learning, many researchers have tried to apply learning-based method to resolve this problem.

Figure 1: Traditional pipeline [2]

In line with those researches, recently we have found a paper contributed by Karadeniz, A.S., Erdem, E. and Erdem, A. The paper claims to achieve SOTA result by using coarse-to-fine architecture, resulting in much sharper and higher quality images. Motivated by this result, we have built a model based on this research paper and achieved some results, which we would like to introduce later in this post.

The implementation source code has been submitted to https://paperswithcode.com/ as a baseline if someone would like to improve further. https://paperswithcode.com/paper/burst-denoising-of-dark-images

Figure 2. Our implementation has been submitted to PaperWithCode

Summary

Figure 3. Network architectures of the proposed (a) single-frame coarse-to-fine and (b) set-based burst models. [1]

According to the authors, the whole pipeline includes two models, one is Coarse Network and another is Fine Network, the Set-based Network is used to improve color accuracy. In the context of this implementation, we stop at building a single-frame model including Coarse and Fine Networks (not included the Set-based network).

Briefly, the Coarse Network is trained using raw input images with 4 channels (RGGB), the output of Coarse Network is a raw (denoised) image with 4 channels as well. This output plus the raw input x, and the noise which is described as the difference between input x and output y of Coarse Net, are concatenated to create an input image of 12 channels. This combination will then be used to train the Fine Network, this is supposed to further refine the output RGB image.

Coarse and Fine Networks use the same Unet-based architecture and use L1 as a loss function.

Result

The result is measured using PSNR and SSIM metric[3], evaluation is done on Sony subset of SID dataset. In the paper, PSNR and SSIM are 29.43 and 0.891, respectively.

Compare to that result, our implementation achieved a lower result with PSNR of 28.91 and SSIM of 0.79, but this result is also higher than previous researches on the same problem.

Figure 4. Good results from the model

Limitations

Besides good results from the model (as shown in Figure 4), we also see failure in some cases when the image is captured under the extremely dark environment with pretty high ISO level (above 10000 from our observation), see Figure 5, this problem has also been reported in the paper by the authors (“our results are still not perfect and there is room for improvement, especially for the scenes with extremely dark illumination conditions”). As can be seen, the model failed to preserve the color and the detail of the image.

Besides, the prediction image also has higher level of green color (Figure 6) which we haven’t figured out the root causes.

Figure 5. Image taken under extremely low-light conditions can not be properly recovered

Figure 6. Prediction image with high level of green color

Conclusion

In summary, based on the paper Burst Denoising of Dark Images, we have created a base-line Coarse-to-Fine model to denoise images taken from low-light conditions (https://paperswithcode.com/paper/burst-denoising-of-dark-images ). The model works well except when the images are taken under extremely dark lighting condition, in such case, we found that the ISO level of those images is extremely high(above 10000), this could be the cause for the model to be failed when trying to recover the original image’s detail. The produced images from our model are good but there are cases in which the color is not preserved correctly, using Set-based model may help fix this issue according to the authors but we haven’t tried in this work. You could try that and show the result for comparison.

Thank you for reading!

Disclaimer: The submitted source code is an unofficial implementation of the paper and may not be correct with the original idea of the authors, it is just our try to reproduce the result from the authors’ research.

References:

[1] Karadeniz, A.S., Erdem, E. and Erdem, A., 2020. Burst Denoising of Dark Images. arXiv preprint arXiv:2003.07823.

[2] Chen, C., Chen, Q., Xu, J. and Koltun, V., 2018. Learning to see in the dark. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3291–3300).

[3] Hore, A. and Ziou, D., 2010, August. Image quality metrics: PSNR vs. SSIM. In 2010 20th international conference on pattern recognition (pp. 2366–2369). IEEE.