Super-Resolution with Generative Adversarial Network (GAN)

Introduction

When we try to show a low-resolution image on a big screen the image may become blurry; therefore, we need a method to up-scale that image without losing quality.

Super Resolution (SR) is a software technique that enhances the resolution of the image.

Summary

There are several approaches to this problem. These approaches illustrated below:

Figure 1: Main category of approaches

Figure 2: Example-based categories

Figure 3: LEFT: low resolution, RIGHT: SR output

Note: This document only represents Deep Learning approaches.

Deep learning

Deep Learning is a subfield of Machine Learning concerned with algorithms inspired by the structure and function of the brain called artificial neural networks.

Figure 4: artificial and biological Neuron. In deep learning, a single neuron called a unit

Figure 5: Deep learning illustration

Approaches

· CNN: Convolutional Neural Network

· ResNet: Residual Neural Network

· GAN: Generative Adversarial Neural Network

Convolutional Neural Network:

This is one of many architectures of neural network which widely used in image and signal processing (Figure 6).

A special layer called Convolutional layer used in the early layers which are the core of this architecture (Figure 7).

There is also a pooling layer as shown in Figure 6. The pooling layer replaces the output of the network at certain locations by deriving a summary statistic of the nearby outputs. This helps in reducing the spatial size of the representation, which decreases the required amount of computation and weights. The pooling operation is processed on every slice of the representation individually (Figure 8).

Figure 6: CNN architecture

Figure 7: An illustration of a convolutional layer

Figure 8: Pooling layer illustration

ResNet:

This is another popular network that uses skip connections in its architecture as shown in Figure 9. These skip connections help the network to reduce the chance of overfitting.

Figure 9: ResNet

In each layer we have:

Where a[l] is the activation of layer “l” and g is the activation function (e.g. Relu).

GAN:

Generative Adversarial Network is a brand new architecture that used in many deep learning tasks such as Image to Image Translation.

Figure 10: GAN architecture

Generative modeling involves using a model to generate new examples that plausibly come from an existing distribution of samples, such as generating new photographs that are generally similar but specifically different from a dataset of existing photographs.

A GAN is a generative model that is trained using two neural network models. One model is called the “generator” or “generative network” model, which learns to generate new plausible samples. The other model is called the “discriminator” or “discriminative network” and learns to differentiate generated examples from real examples.

The two models are set up in a contest or a game (in a game theory sense) where the generator model seeks to fool the discriminator model, and the discriminator is provided with both examples of real and generated examples.

In the end, if we want to compare different approaches we would have: