Deep Learning in 5 minutes Part 3: Discriminative vs Generative Models

In machine learning field is common to talk about discriminative and generative models. Before to talk about autoencoders and GANs it is important to make clear some concepts.

Discriminative Models

Discriminative models. Source: Google Images

A Discriminative model ‌models the decision boundary between the classes and learns the conditional probability distribution p(y|x) [1]. Some examples of discriminative models are logistic regression, SVMs, ANN, KNN and Conditional Random Fields.

Generative Models

Generative Models. Source: Google Images

A Generative Model ‌explicitly models the actual distribution of each class and learns the joint probability distribution p(x,y). It predicts the conditional probability with the help of Bayes Theorem [1]. Some examples of generative models are Näive Bayes, Gaussians, HMM, Mixture of Gaussians, Bayesian networks, Markov Random Fields and Mixture of multinomials.

Generative — Discriminative Pairs

While Näive Bayes and Logistic Regression are a pair for classification; HMM and CRF are a corresponding pair for sequential data.

To recap

Generative classifiers

  • Assume some functional form for P(Y), P(X|Y)
  • Estimate parameters of P(X|Y), P(Y) directly from training data
  • Use Bayes rule to calculate P(Y |X)

Discriminative Classifiers

  • Assume some functional form for P(Y|X)
  • Estimate parameters of P(Y|X) directly from training data [1].

Resources

  1. Generative vs Discriminative Models http://bit.ly/2Awl5dx
  2. Machine Learning: Generative and Discriminative Models — CEDAR http://bit.ly/2AwYQVc
  3. Generative and Discriminative Models http://bit.ly/2AsPtWh