Image Restoration Using Morphological Opening-Closing Network
Image Restoration by Learning Morphological Opening-Closing Network
Ranjan Mondal, Moni Shankar Dey & Bhabatosh Chanda, Mathematical Morphology-Theory and Applications 4 (1), 87–107.
DOI : https://doi.org/10.1515/mathm-2020-0103
Code : https://github.com/ranjanZ/Mophological-Opening-Closing-Net
Introduction
Outdoor images are used to train algorithms for various tasks, such as autonomous navigation and surveillance. As the imaging systems interact with their environment, they are exposed to a wide variety of weather conditions such as rain, haze or fog. This may result in colour or contrast degradation, which produces a low-quality image. These type of images need to be restored to their original state by removing noises before they are deemed suitable to be used in computer vision-based tasks. Image de-hazing and de-raining are highly challenging tasks and are some of the most widely researched topics in the field of image restoration.
Classical computer vision (CV) algorithms rely on developing handcrafted features, borrowing concepts from diverse fields such as set theory, statistical physics and signal processing, for image analysis. Deep learning (DL) based methods are widely used nowadays to solve challenging problems in image processing such as segmentation, classification and detection. Unlike traditional CV techniques, high-level abstract features relevant to the dataset are extracted automatically. Although DL algorithms achieve higher accuracy, the higher level abstract features are often hard to interpret, which results in their black-box like behavior.
There is ongoing research to design a high performing model with better transparency, combining the advantages of both CV and DL techniques. In this article, we briefly describe our paper [1] in which we use the novel Opening-Closing network, a type of morphological neural network, for image de-hazing and de-raining.
Mathematical Morphology
Morphological operations are based on set theory, which provides it with a rich mathematical framework. They are suitable for problems involving an object’s shape and are used widely for image analysis. The basic morphological operations are dilation and erosion. In an image, the dilation operation adds pixels to the object’s boundaries, while erosion removes pixels from borders.The performance of morphological operations depends on the size and shape of the structuring element (SE), which in turn is dependent on the problem at hand.
In binary images, the pixels belong to either foreground or background and belong to sets in 2D euclidean space. On the other hand in grayscale images, each pixel has an intensity along with their spatial coordinates, and are part of a 3D set, called the umbra of image. One can visualize the grayscale image surface composed of numerous peaks and valleys representing bright and dark regions. The grayscale morphological operations take into account the slope and curvature for image analysis. In grayscale morphology, the erosion and dilation operations are defined using Min and Max operations over SE. Complex morphological operations such as closing, opening and top hat operations are implemented by arranging dilation and erosion in a particular sequence.
Opening Closing Network
The performance of morphological operations depends on the shape and size of SE. In practice, it is quite challenging to determine the correct sequence of morphological operations and their associated SEs for the task at hand. Over the years, several techniques have been developed to learn from SE from the data. Convolution filters and SE are neighborhood operations which consider the spatial relationship between pixels. Seeing the similarity, Mondal et al. [2] proposed a 2D morphological network for grayscale image de-raining. The network is end to end trainable and is composed of several morphological operations which are linearly combined. The SE is piece-wise continuous in nature and is learnt using backpropagation.
In our paper, we extend the above work from grayscale to colour images. We design an Opening Closing network, composed of sequence of dilation and erosion operations, for image de-hazing and de-raining. This network has several advantages over conventional convolution based networks:
- Grayscale morphological operations are based on Min-Max algebra, due to which there is an inherent non-linearity in the system. This eliminates the need for any activation function in the network.
- Dilation, erosion operations produce more complex decision boundaries, which results in the network having fewer parameters, while still able to extract high-level features.
- As the model employs dilation and erosion operations, one is able to interpret the feature maps, which makes the model transparent.
Image De-Raining
Rain droplets in an image are considered as bright noise. There are variations in raindrop size, density and lighting condition, all of which need to be taken care of. To remove them, we design a network containing a sequence of opening and closing operations so that it performs like an Alternate Sequence Filter (ASF). The image similarity can be calculated using the Structural Similarity Index Metric (SSIM), which uses the local statistics from a patch. Hence to train the network, we use SSIM as a loss function, in the form of DSSIM (i.e. 1-SSIM). It is observed that the network using only closing operations is better suited at removing raindrops as it is a bright noise. But generally we prefer using both opening and closing operations in the network.
Image De-Hazing
Light rays travelling through a turbid medium undergo a phenomenon known as scattering, wherein the light is scattered in different directions. This occurs is due to its interactions with particles, such as dust and aerosols floating the medium. Haze occurs when the concentration of these particulate matters exceed a certain threshold. One can physically model the effect of haze at location x as
I(x) = t(x)J(x) + (1-t(x))A
Here the observed intensity (I) depends on the effect of transmittance (t) on non-hazy image(J) and airlight (A). The second term in the equation represents the contribution of the environment and is expressed as K(x). Using the opening-closing network, we determine the parameters t(x) and K(x), so that we get the haze-free image using the Haze equation. We use DSSIM loss function to train the network for image de-hazing.
Results
The Opening-Closing network is implemented in Keras with Tensorflow backend. We use the benchmark Rain and O-HAZE & NYU D-Hazy datasets for training the de-raining and de-hazing networks, respectively. For quantitatively evaluating the model performance we use Peak Signal to Noise Ratio (PSNR) and SSIM.
Image De-Raining
Image De-Hazing
Conclusion
To summarise, we propose a novel Opening-Closing network combining the strengths of both morphological operations with learning-based methods. Our network has no non-linear activation function and very low number of parameters compared to traditional networks. The presence of opening and closing feature maps helps in making the model interpretable and transparent.
In our work, we have used a very simple architecture for image de-hazing and de-raining to demonstrate its generalization ability. Even with a small number of parameters and unoptimized architecture, it is able to give results comparable to the state of the art methods.
References
[1] R. Mondal, M.S. Dey and B. Chanda, Image Restoration by Learning Morphological Opening-Closing Network (2020), Mathematical Morphology-Theory and Applications 4 (1), 87–107
[2] R. Mondal, P. Purkait, S. Santra and B. Chanda, Morphological Networks for Image De-raining (2019), International Conference on Discrete Geometry for Computer Imagery, 262–275
