GANs — A Brief Overview (Part 1)
This post presents the basic notions that involve the concept of Generative Adversarial Networks — GANs.
In this article, you will find:
- Background of some previous concepts,
- A brief introduction to Generative Adversarial Networks,
- Network design and cost function,
- Training, and
- Mind map with a GAN’s summary.
Background
A brief definition of some concepts is presented, such as supervised and unsupervised learning, and discriminative and generative models.
Supervised learning
Learn to map a function ŷ=𝑓(𝑥) given labeled data y.
- Examples: Classification algorithms (SVM), regression algorithms (linear regression).
- Applications: Object detection, semantic segmentation, image captioning, etc.
Unsupervised learning
Learn the underlying structure of the given data, without specifying a target value.
- Examples: Clustering algorithms (k-means), generative models (GANs)
- Applications: Dimensionality reduction, feature learning, density estimation, etc.
Discriminative models
Most of the supervised learning algorithms are inherently discriminative.
The Discriminative model allows us to evaluate the conditional probability 𝙿(𝑦|𝑥). Learns a function that maps the input x to an output y.
- Examples: Logistic regression, Support Vector Machines, Neural networks.
Generative models
Most of the unsupervised learning algorithms are inherently generative.
The Generative model can allow us to evaluate the joint probability 𝙿(𝑥, 𝑦). Tries to learn a joint probability of the input x and the output y at the same time.
- Examples: Latent Dirichlet allocation, Restricted Boltzmann machine, Generative adversarial networks.
Generative Adversarial Networks — GANs
Generative Adversarial Networks or GANs is a framework proposed by Ian Goodfellow, Yoshua Bengio, and others in 2014.
GANs are composed of two models, represented by an artificial neural network:
- The first model is called a Generator, and it aims to generate new data similar to the expected one.
- The second model is named the Discriminator, and it seeks to recognize if an input data is ‘real’ — belongs to the original datasets — or if it is ‘fake’ — generated by a forger.
Network Design
- The input to the generator 𝐺 is a series of randomly generated numbers called latent space. It tries to produce data that come from some probability distribution.
- 𝐺 takes random noise as input, then runs that noise through a differentiable function to transform the noise and reshape it to have a recognizable structure.
- The output of 𝐺 is a realistic image. Without training, the generator produces garbage images only.
- 𝐷 is a classifier trained using supervised learning. It classifies whether an image is real (1) or is fake (0).
Cost Function
Minimax objective function (Value Function or Cost Function of Minimax Game played by Generator and Discriminator):
- 𝐷⒟ wants to maximize objective such that 𝐷(𝑥) is close to 1 (real), and 𝐷(𝐺(𝑧)) is close to 0 (fake).
- 𝐺⒢ wants to minimize objective such that 𝐷(𝐺(𝑧)) is close to 1 (discriminator is fooled into thinking generated 𝐺(𝑧) is real).
Two-player game
The Generator 𝐺 (forger) needs to learn how to create data in such a way that the Discriminator 𝐷 isn’t able to distinguish it as fake anymore. The competition between these two teams improves their knowledge until the 𝐺 succeeds in creating realistic data.
- 𝐺 try to fool discriminator by generating real-looking images.
- 𝐷 try to distinguish between real and fake images.
As a result, the Discriminator 𝐷,
- 𝐷 is trained to classify the input data as either real or fake correctly.
- 𝐷 weights are updated as to maximize the probability that any real data input x is classified as belonging to the real dataset.
- The loss/error function used maximizes the function 𝐷(𝑥), and it also minimizes 𝐷(𝐺(𝑧)).
and the Generator 𝐺,
- 𝐺 is trained to fool the Discriminator by generating data as realistic as possible.
- 𝐺 weights are optimized to maximize the probability that any fake data is classified as belonging to the real dataset.
- The loss/error function used for this network maximizes 𝐷(𝐺(𝑧)).
Training
We train the Discriminator and the Generator in turn in a loop as follows:
- Set the Discriminator trainable
- Train the Discriminator with the real data and the data generated by the generator to classify the real and fake data.
- Set the Discriminator non-trainable
- Train the Generator as part of the GAN. We feed latent samples into the GAN and let the Generator to produce data and use the Discriminator to classify the data.
It’s essential to choose an excellent overall architecture,,
- That both the Generator and the Discriminator have at least one hidden layer.
- Two simultaneous optimizations.
- We define a loss for the G and D. Minimize the loss for the D while simultaneously use another optimizer to minimize the loss for the G.
For those looking for all the articles in our GANs series. Here is the link.
References
Related papers
- Generative Adversarial Networks
- Unsupervised Representation Learning With Deep Convolutional
- Conditional Generative Adversarial Nets
- Semi-Supervised Learning with Context-Conditional Generative Adversarial Networks
- Wasserstein GAN
- Least Squares General Adversarial Networks
- Generative Adversarial Networks: A Survey and Taxonomy
- Recent Progress on Generative Adversarial Networks (GANs): A Survey
