Generative VS Discriminative Models

I would like to start this write up with a story.

Story:

One fine day, The father takes two of his kids (Kid A and Kid B) to a zoo. This zoo is a very small one and has only two kinds of animals say a lion and an elephant. After they came out of the zoo, the father showed them an animal and asked both of them “is this animal a lion or an elephant?”

The Kid A, the kid suddenly draw the image of lion and elephant in a piece of paper based on what he saw inside the zoo. He compared both the images with the animal standing before and answered based on the closest match of image & animal, he answered: “The animal is Lion”.

The Kid B knows only the differences, based on different properties learned, he answered: “The animal is a Lion”.

Here, we can see both of them is finding the kind of animal, but the way of learning and the way of finding answer is entirely different. In Machine Learning, We generally call Kid A as a Generative Model & Kid B as a Discriminative Model.

In General, A Discriminative model ‌models the decision boundary between the classes. A Generative Model ‌explicitly models the actual distribution of each class. In final both of them is predicting the conditional probability P(Animal | Features). But Both models learn different probabilities.

A Generative Model ‌learns the joint probability distribution p(x,y). It predicts the conditional probability with the help of Bayes Theorem. A Discriminative model ‌learns the conditional probability distribution p(y|x). Both of these models were generally used in supervised learning problems.

If you are unaware of joint probability & conditional probability, Please read it here.

In Math,

Generative classifiers

  • Estimate parameters of P(X|Y), P(Y) directly from training data
  • Use Bayes rule to calculate P(Y |X)

Discriminative Classifiers

  • Estimate parameters of P(Y|X) directly from training data

Examples:

Generative classifiers

  • Bayesian networks
  • Markov random fields
  • ‌Hidden Markov Models (HMM)

Discriminative Classifiers

  • Scalar Vector Machine
  • ‌Traditional neural networks
  • ‌Nearest neighbour
  • Conditional Random Fields (CRF)s

Questions:

  1. What are the problems these models can solve?
  2. Which model learns joint probability?
  3. Which model learns conditional probability?
  4. What happens when we give correlated features in discriminative models?
  5. What happens when we give correlated features in generative models?
  6. Which models works very well even on less training data?
  7. Is it possible to generate data from with the help of these models?
  8. Which model will take less time to get trained?
  9. Which model will take less time to predict output?
  10. Which model fails to work well if we give a lot of features?
  11. Which model prone to overfitting very easily?
  12. Which model prone to underfitting easily?
  13. What happens when training data is biased over one class in Generative Model?
  14. What happens when training data is biased over one class in Discriminative Models?
  15. Which model is more sensitive to outliers?
  16. Can you able to fill out the missing values in a dataset with the help of these models?

Here is a nice paper by Professor Andrew NG on Generative and Discriminative Models, Read it if you want to go deep.

--

--

Machine Learning Engineer

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store