Super Resolution and its Recent Advances in Deep Learning — Part 1
Hi and welcome to this series on Super-Resolution. This is mainly aimed at Machine Learning and Computer Vision enthusiasts with little or no background in these fields. In this part, I will briefly introduce the concept of Super-Resolution and its confounding applications in various fields. I will also discuss a few initial algorithms which paved the way to current progress in this domain.
Super Resolution (SR)
Remember the times when you thought you clicked a wonderful picture on your phone, but then you tried to view it on your laptop and it looks blurry and, not so wonderful anymore? Or the times when images looked sharper on a phone rather than on a tablet? This happened because you were trying to enlarge an image that belonged to a phone screen (low resolution — LR) to fit the size of a laptop screen (high resolution — HR).
Similar to how alphabets form words, a number of small colored squares called pixels are arranged in an organized fashion to create images. A ‘high resolution’ image requires more pixels to represent sharper details and better texture. The process of generating a high-resolution image from it’s low-resolution counterpart is termed as super resolution.
Why super resolve an image?
Why don’t we just capture a high-resolution image instead of going through all this technical stuff? The answer is the same as why we don’t go and buy a DSLR when it’s clearly better than a phone camera. Yes, capturing a high-resolution image is expensive and is sometimes impossible. I will list a few applications here.
(i) Medical Image Processing
Medical images (like MRI, CT, PET, etc) are extremely valuable for the diagnosis of a disease. Some times these scans don’t help due to resolution limitations. SR-techniques have been used to generate better images.
(ii) Video Enhancement and Optical Image Stabilization
Super-resolution techniques have been used on low-resolution images in videos (Standard definition — SDTV) to create close to High Definition Videos (HDTV) to serve HD screens. Super-resolution can also be used to reduce blurring caused due to motion associated with the camera. To resolve such issues, SR-techniques are hoped to be employed in phones and tablets soon.
(iii) Surveillance
Surveillance cameras have become extremely common for traffic and security supervision. Needless to say, it’s impossible to install high-resolution cameras everywhere. Since videos contain more data (requiring more computational power) than images and are subject to camera motion artifacts, super-resolution in this domain gets even more complex. Most of the video super-resolution approaches are still in the research stage to be implemented in real life.
(iv) Astronomical Observation
Astronomical devices’ resolution is limited by its hardware. In such cases, Super-resolution is a boon to researches exploring and studying outer space.
How do we super resolve then?
If the number of pixels is the only issue, why don't we just take a low-resolution image (100x100 pixels), add more pixels to it and color each pixel based on its neighboring pixels? This is indeed the main idea behind techniques like Nearest-neighbor upsampling, Bilinear interpolation, and Bicubic interpolation. They differ in the methodology used to fill the extra pixel values.
However, super-resolution didn’t turn out to be that simple. These plain interpolations created either pixelated or blurry images. What we desire are sharper details and better texture. This is when deep learning and Generative adversarial networks came to the rescue. To fill the pixel values more strategically, we need to observe, study, and gain more information about the general characteristics of a particular style of images say knee MRI scans. This is exactly what neural networks do, they study the training data (set of HR-LR MRI image pairs) and tune their network parameters to adapt to the data properties. Now, if we provide a new knee MRI scan to our network, it knows what to do to return us the super-resolved scan. Once the network is trained for 10x10pixels(LR) to 20x20pixels(HR) conversion, a new image of 20x20pixels(LR) can be passed through the network to obtain 30x30pixels(SR).
A Deep learning network can be looked at as a highly complex non-linear equation which is a function of input variables (in our case, pixel values of an image) designed to serve our purpose.
Stay tuned to explore various deep learning approaches developed in recent times for Super-Resolution in my upcoming parts of this series :)
References
- Deep Learning for Image Super-resolution: A Survey
- Image super-resolution: The techniques, applications, and future
Links to all the parts:
- Super Resolution and its Recent Advances in Deep Learning — Part 1
- Super Resolution and its Recent Advances in Deep Learning — Part 2
Stay tuned, more parts coming soon :)
