Linear Discriminant Analysis
Linear Discriminant Analysis is one of the commonly used supervised technique for dimensionality reduction. It is also used in classification problems and for data visualizations.
Dimensionality Reduction is the transformation or projection of data from higher-dimensional space to lower-dimensional space.
How is LDA different from PCA?
The major distinction between LDA and PCA is that, LDA focuses on finding the axes that maximize the separation between multiple classes.
Criteria used by LDA to create new axis are:
- Maximize the distance between the classes.
- Minimize the variation within each class.
In simple terms, this newly generated axis by LDA increases the separation between the data points of the two classes as indicated by green line in the above figure.
Drawbacks of LDA
Linear Discriminant Analysis fails when the mean of the distributions are shared, as it becomes impossible to find a new axis that makes both the classes linearly separable. In such cases, we use non-linear discriminant analysis.
Extensions to LDA
In order to deal with non-linearly separable space we use the following extensions of LDA
- Quadratic Discriminant Analysis (QDA): Each class uses its own estimate of variance (or covariance when there are multiple input variables).
- Flexible Discriminant Analysis (FDA): Where non-linear combinations of inputs is used such as splines.
- Regularized Discriminant Analysis (RDA): Introduces regularization into the estimate of the variance (actually covariance), moderating the influence of different variables on LDA.
Implementation: Refer the following link for Python and R implementation of Linear Discriminant Analysis:
