About Generative and Discriminative models

Have you ever heard about generative or discriminative models? Do you know what they are? Do you want to know the difference between them? Which one is better?

Discrimina…what?

Imagine that our task is to classify the input data x into labels y. We can use Machine Learning to do this, but we have two different approaches:

• Generative models: Generative models describe how data is generated using probabilistic models. They predict P(y|x), the probability of y given x, calculating the P(x,y), the probability of x and y.
• Discriminative models: A discriminative model does not care how the data is generated. Here we just care about P(y|x).

“Just fuzzy words so far.”

In practice

For example, imagine we are trying to classify dogs and cats based on the animal weight and height using a generative model. We will have to compute the following probabilities for each data point:

• P(cat,weight)
• P(cat,height)
• P(dog,weight)
• P(dog,height)

Imagine that we have 1,000 data points to train our model. This means that at least we will need to compute 4,000 probabilities. What we are doing computing these probabilities is something similar to this:

In here, we try to know what dogs and cats are. The model gets their features (weights and heights) generates two probability distributions based on the features and the classes. So, when we ask the model if a certain animal is whether a dog or a cat, it can map the features to the classes and see which one is more likely.

When the generative model has the probabilities distribution, it draws a decision boundary based on them.

However, imagine we choose to use a discriminative model. Now, we just need to compute P(y|x) for each data point. This means that we just need to calculate 2,000 probabilities if the data set has 1,000 data points . One way to think about discriminative models is that they rely on the differences between classes given by the features.

In the dogs and cats example, a discriminative model would try to draw a decision boundary that separates the cats and dogs. Then, to classify a new animal as either an cat or a dog, it checks on which side of the decision boundary it falls, and makes its prediction accordingly.

The above picture shows that a discriminative models does not care about describing what cats or dogs are, it is just trying to segregate both classes.

Key difference

The decision boundary is defined by those points which have the same probabilities to be classified as dog or as cat. But the decision boundary is different for each model, and it is the key.

In the generative model, the decision boundary is defined by P(x,y). Imagine that we did not have any fat cat in our training data, and we ask to our model if a very fat cat is a cat or a dog. The model computes the probability of a fat cat and returns the value p, which is very small, but at the same time computes the probability of a dog being the size of a cat and gets the same small value p. At this point, the algorithm does not know how to classify this data point because it is on the decision boundary. So mathematically, the decision boundary is where P(x,cat) = P(x,dog).

In the discriminative model, the decision boundary is defined by P(y|x). Here, the decision boundary is where P(dog|x) = P(cat|x).

Generative vs Discriminative

As everything in Machine Learning, no one is better than the other. It all depends on the task you are working on.

For example, imagine you have to classify a speech to a certain language. Which approach would you take?

• Learning each language, and then classifying it using the knowledge you just gained
• Determining the difference in the linguistic models without learning the languages, and then classifying the speech.

Although it is true that discriminative models are more popular, generative models can perform very well on several problems.

Pros and Cons

Generative models

Pros:

• Good at unsupervised machine learning
• We get the underlying idea of what the class is built on

Cons:

• Very computational expensive

Discriminant models

Pros:

• Need less data
• Computationally cheaper

Cons:

• Not useful for unsupervised learning
• Can be more difficult to interpret

Examples

Generative models examples

• Naive Bayes
• Gaussian mixture model
• Hidden Markov Models (HMM)

Discriminative models examples

• Logistic regression
• SVM
• Neural Networks

Open question

Are you a cat or a dog person?