Breaking Through the Darkness: How to Enhance Low-Light Images with Deep Learning Techniques
In photography, a professional photographer can change his lens, adjust the amount of light exposure (aperture), shutter speed as well as light sensitivity (ISO) to compensate for dark or low-light conditions to capture the perfect image. For static cameras however, particularly those used in public security areas, it is neither practical nor cost-effective to have large-scale deployment of cameras that can adjust for different lighting conditions to do the same. So how can we enhance images that have been taken in unfavourable and low-light conditions? In this article, we will discuss Deep Learning techniques that can be used to process and enhance low-light images.
How do we assess low image quality for enhancement?
Low-light image enhancement is simply the enhancement of images captured in a low ambient light environment, to improve image visual clarity as shown below.
(Observe the vehicles and pedestrians are visible after Low-Light Enhancement)
[Image credits: ExDark Dataset¹]
Before going into the process of low-light image enhancement, it is important to first determine what are the issues or causes of low-light images. Causes of low-light conditions can be due to: (1) insufficient or absent light source; (2) uneven illumination caused by back-light and shadows; or (3) incorrect configuration of the camera and limitations of photo-sensors in capturing the full range of light and colours (or underexposed images). Examples below:
[Image credits: ExDark Dataset¹]
First, we need to quantify what is a ‘low-light’ image. A histogram can be used to visualise the luminance channel of image pixels. The luminance histogram (below) shows the distinct characteristics of images captured in poorly illuminated environments versus those captured in well illuminated environments.
[Image Credits: Images from ExDark Dataset¹, Histograms generated by HTX S&S CoE]
The histogram of the low-light image (on the left) shows two observations. First, the pixel values are concentrated within a narrow range of darker tones and the entire histogram is unevenly distributed. Second, the histogram displays a singular dominant peak, indicating a lack of tonal variation. In contrast, a non low-light image (on the right) exhibits a histogram with broad distribution across the tonal range. The histogram is also characterised by multiple peaks and troughs that are distinctly separated from one another.
We think a method to assess image quality is essential as a precursor to low image enhancement. Being able to assess image quality is a prerequisite for effective image processing, enhancement, and correction. To be able to quantifiably assess images with low-light problems means that we will not have to deal with the subjective perception of visual clarity. This can translate to more reliable and repeatable image processing outcomes.
So, we developed a method using Support Vector Machines (SVM) with input features derived from the Cumulative Distribution Function (CDF) obtained from the histogram of images. An example of a CDF histogram below.
[Image credits: Graphs generated by HTX S&S CoE ]
The CDF of an image histogram is a mathematical function that transforms the histogram into a representation of the cumulative probability of pixel intensity values in the image. CDF is a superior feature input for SVM classification compared to histograms, as it goes beyond the ability to represent key statistics such as minimum, maximum, median, and percentiles. CDF offers a more comprehensive representation of the data, enabling a better representation of data outliers and distribution. This makes it more effective in image classification tasks, particularly when dealing with low-light image datasets. The examples below show that histograms are susceptible to misrepresentation of data outliers, especially when the bin size is incorrect.
In the histogram on the left, the outlier is not observable (due to binning bias) but revealed in the histogram in the middle. For CDF, the outlier is more prominently represented as at tail of CDF curve.
[Image credits: Graphs generated by HTX S&S CoE ]
CDFs can eliminate the issue of incorrect bin size (also known as binning bias) and offer a more accurate method to identify and analyse outliers that typically reside in the tails of the data distribution (as shown below).
[Image Credits: HTX S&S CoE]
CDF is a powerful tool for analysing data distributions and uses statistical tests such as the Kolmogorov–Smirnov (K-S) test to measure the similarity between different classes. Unlike histograms, which are limited to visual comparison, the K-S test uses the CDF to calculate the distance measure between two classes for accurate and quantifiable comparison of data.
In our experiment, we used a dataset of 11,000 open-source images (AMOS², LOL³) to test our hypothesis. Our hypothesis posited that using CDF as input features for a Support Vector Machine (SVM) would yield superior performance in classifying low-light images as compared to using histograms. The results of the validation test supported our hypothesis, showing that the SVM trained with CDF (SVM-CDF) outperformed the SVM trained with histograms (SVM-HIST) in overall prediction accuracy.
The SVM-CDF method demonstrated superior classification accuracies, as evidenced by the macro-averaged F1-score of 97.1%, compared to SVM-HIST’s macro-averaged F1-score of 79.7% in the table below.
[Image Credits: HTX S&S CoE ]
Further analysis of SVM-HIST’s accuracy performance revealed that the model was susceptible to higher rates of false positives, resulting in lower precision but higher recall. It is evident that CDF is a more effective input feature for the automatic detection of low-light images.
Low-Light Image Enhancement Methods
There are many methods developed to enhance low-light images. They can be broadly categorised as follows:
- Histogram Equalisation Methods: This is a technique in computer image processing that improves global contrast of an image by spreading out the intensity values, resulting in a more balanced distribution of pixel intensities. This can result in an image with improved contrast, as it brings out details in both the light and dark regions of the image. It can, however, lead to over-enhancement in certain areas of the image without considering local variations. In regions where there are already high levels of contrast or significant intensity variations, Histogram Equalization can amplify these differences excessively. As a result, those areas may appear overly bright or dark, losing the balance and natural appearance of the image.
- Retinex Methods: These are based on a theoretical model that simulates the human visual system’s ability to perceive images under different lighting conditions. A common approach in Retinex methods is to separate the illumination component from the reflectance component, and then apply appropriate algorithms to enhance the reflectance component while preserving the illumination component. This results in improved image quality, with enhanced colour accuracy, contrast, and detail.
- Deep Learning Methods: These techniques use Deep Neural Network architectures to learn complex features and patterns that are specific to low-light conditions. After training, the neural network can enhance the quality of low-light images by applying learned transformations or adjustments to the input images.
In this article, we will focus on the use of Deep Learning methods such as ZeroDCEs (Zero-Reference Deep Curve Estimation) & its variants for enhancing low-light images. According to a research paper⁴ published in IEEE Computer Society, ZeroDCE has shown better results than other existing Zero-Reference methods. The difference is that in traditional Zero reference models, the darkest and brightest pixels are often used as reference points. ZeroDCE takes a different approach by using a deep learning-based model to estimate a curve that enhances the image based on its characteristics, without relying on any specific reference points. This prompted us to investigate ZeroDCE & its variants further and explore adapting it for our needs with our own curated set of low-light images.
DN-ZeroDCE , a noise-reduction variant of ZeroDCE
ZeroDCE, and its improved version ZeroDCE++ [ZeroDCEs], are cutting-edge low-light enhancement models that have garnered significant attention in the field of deep learning image processing. These models were developed by researchers and employ a unique approach to low-light enhancement. Conventional methods require paired datasets (consisting of low-light and normal-light image pairs), whilst ZeroDCEs use a Convolutional Neural Network (CNN) composed of seven convolution layers with non-reference loss functions to estimate the best parameters for the light enhancement (LE) curve of an image.
What makes ZeroDCE approach stand out from other low-light enhancement models is that the model training can be done without the need for a paired dataset. This makes it highly desirable in the field, as it mitigates the difficulty of collecting such data and allows for easier training of the model. The approach is not only accurate and robust but also capable of processing images in real-time (at a framerate of ~20 FPS or better, for 1290 x 1080 images on RTX 3090 GPU card).
In our work, we focused on improving ZeroDCE’s ability to effectively de-noise input images and achieve superior enhancement results. We added an autoencoder module to the front of the ZeroDCE network and developed the architecture for De-Noise ZeroDCE (DN-ZeroDCE) as shown below.
[Image Credits: HTX S&S CoE ]
In our experiments, DN-ZeroDCE has been effective in removing noises associated with low-light images. The original ZeroDCE model relies on a set of carefully crafted non-reference loss functions. These loss functions include colour constancy loss, exposure loss, illumination smoothness loss, and spatial consistency loss. DN-ZeroDCE is an improved version of Zero-DCE that incorporates an additional non-reference noise loss function.
To facilitate noise reduction during model training, we introduced the Noise Level Function (NLF)⁵ as the non-reference noise loss function for DN-ZeroDCE. The NLF serves two purposes: (1) To preserve the model’s zero-reference characteristic, and (2) to train the network’s ability to do noise removal from the enhanced images. The NLF measures the standard deviation of noise in relation to image intensities of both the original image and the enhanced image.
The NLF noise loss function can be represented as:
[Credits: Ce Liu, Freeman, W. T., Szeliski, R., & Sing Bing Kang⁵]
Our DN-ZeroDCE approach can improve the enhanced image by reducing the noise levels, indicating the model’s effectiveness in noise reduction in images. This can be seen below from the output artefacts from both ZeroDCE and DN-ZeroDCE models.
[Image Credits: HTX S&S CoE ]
DN-ZeroDCE is better able to suppress noise artifacts arising from image enhancement (or the original noise present in the images). You can see the visible noise speckles in the ZeroDCE enhanced image on the left but not so much of it on the right. This enhancement is important as it frees the output images of unwanted noise and distortion that can compromise their utility in real-world applications.
Our experiments have demonstrated that the DN-ZeroDCE can achieve better image enhancement results in terms of peak signal–to-noise ratio (PSNR), structural similarity (SSIM), and mean absolute error (MAE), as shown in Table 1 below. Incorporating the autoencoder module before the ZeroDCE model has made significant improvements to both PSNR and SSIM indices, while maintaining almost the same inference speed as the original ZeroDCE.
Testing our model on curated low-light image datasets from CCTV
With a curated dataset of low-light images from CCTV, we have retrained and enhanced the model to better handle real-world conditions. The images are from a wide variety of camera models with differing quality of photosensors, resulting in different image captures. These issues are due to deployed camera sensors that have a wider field-of-view, which in turn capture varying lighting conditions within a single image.
[Image Credits: Image from LOL³ ExDARK¹, CCTV images collated by HTX S&S CoE]
Comparing results using the DN-ZeroDCE model on both open-sourced images and in-house curated images, we were able to validate our hypothesis and demonstrate that our approach leads to significant improvements for the 2 image sources in both colour and detail reproduction (as demonstrated below).
[Image Credits: HTX S&S CoE]
A fully automated workflow for low-light image enhancement
Although many low-light image enhancement models are available, they are susceptible to issues such as unrealistic enhancement and colour distortion, leading to sub-optimal output quality. Over-enhancing an image, especially if it is not a low-light image, can lead to the introduction of additional noise and artefacts, ultimately degrading the overall image quality.
[Image Credits: Original image from ExDark¹, enhanced images generated by HTX S&S CoE]
We have developed a low-light enhancement product that does the end-to-end process of automatically correcting different types of low-light problems and generating visually pleasing and natural-looking images.
The workflow is designed to first determine whether an image requires low-light enhancement using the SVM model (discussed in the earlier section). If the image is of low-light and needs enhancement, the image is then passed through the low-light enhancement models.
[Credits: HTX S&S CoE]
After applying the enhancement models, the resulting images are ranked using Image Quality Assessment methods, such as Brisque. From this ranking, we then select the top-N quality image as the output of the low-light enhancement process.
The end-to-end solution provides an efficient and effective way to improve the visual quality of low-light images. By automating the process and utilising the best available models, we can consistently generate natural-looking and visually appealing images. We have since built this as an in-house product known as InXeption, which is an amalgamation of the image enhancement deep learning algorithms as discussed above (and in our previous article on Image Super Resolution) for use within our organisation and selected users.
What we are doing in HTX S&S COE
We are from the Home Team and Science Technology Agency (HTX) in Singapore. Our team is part of the Sense-making and Surveillance Centre of Expertise (S&S COE) within HTX. At HTX S&S COE, we adopt a systematic approach to automate low-light enhancement by benchmarking state-of-the-art techniques and applying optimal enhancement to identified types of low-light problems. We continually iterate our in-house products and solutions for low-light enhancements which involves customisation of models for better performance and deployment in real-world applications.
We hope that you have enjoyed our article on this topic! If you like what we do and want to follow our journey in different AI and sensors engineering work, do subscribe to our medium channel to be kept updated on our projects. Do reach out to me at Benjamin_Cham@htx.gov.sg. if you would like to know more or discuss your ideas on low-light image enhancement.
[1]:Y. P. Loh and C. S. Chan, “Getting to know low-light images with the exclusively dark dataset,” CVIU, vol. 178, pp. 30–42, 2019.
[2]: Jacobs N, Burgin W, Fridrich N, Abrams A, Miskell K, Braswell BH, Richardson AD, Pless R. 2009. The Global Network of Outdoor Webcams: Properties and Applications. In: ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (ACM SIGSPATIAL). 111–120. DOI: 10.1145/1653771.1653789.
[3]: Wei, C.; Wang, W.; Yang, W.; and Liu, J. 2018. Deep retinex decomposition for low-light enhancement. arXiv preprint arXiv:1808.04560.
[4]: In IEEE Transactions on Pattern Analysis and Machine Intelligence (Vol. 44, Issue 12, pp. 9396–9416). Institute of Electrical and Electronics Engineers (IEEE). https://doi.org/10.1109/tpami.2021.3126387 by Li, C., Guo, C., Han, L., Jiang, J., Cheng, M.-M., Gu, J., & Loy, C. C. (2022). Low-Light Image and Video Enhancement Using Deep Learning: A Survey.
[5]: Ce Liu, Freeman, W. T., Szeliski, R., & Sing Bing Kang. Noise Estimation from a Single Image. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition — Volume 1 (CVPR’06).
