Deep Learning in 5 minutes Part 3: Discriminative vs Generative Models
In machine learning field is common to talk about discriminative and generative models. Before to talk about autoencoders and GANs it is important to make clear some concepts.
A Discriminative model models the decision boundary between the classes and learns the conditional probability distribution p(y|x) . Some examples of discriminative models are logistic regression, SVMs, ANN, KNN and Conditional Random Fields.
A Generative Model explicitly models the actual distribution of each class and learns the joint probability distribution p(x,y). It predicts the conditional probability with the help of Bayes Theorem . Some examples of generative models are Näive Bayes, Gaussians, HMM, Mixture of Gaussians, Bayesian networks, Markov Random Fields and Mixture of multinomials.
Generative — Discriminative Pairs
While Näive Bayes and Logistic Regression are a pair for classification; HMM and CRF are a corresponding pair for sequential data.
- Assume some functional form for P(Y), P(X|Y)
- Estimate parameters of P(X|Y), P(Y) directly from training data
- Use Bayes rule to calculate P(Y |X)
- Assume some functional form for P(Y|X)
- Estimate parameters of P(Y|X) directly from training data .