Linear Discriminant Analysis, Explained in Under 4 Minutes
The Concept, The Math, The Proof, & The Applications
Linear Discriminant Analysis (LDA) is, like Principle Component Analysis (PCA), a method of dimensionality reduction. However, both are quite different in the approaches they use to reduce dimensionality. While PCA chooses new axes for dimensions such that variance (and hence the ‘shape’) of the data is preserved, LDA chooses new axes such that the separability between two classes is optimized, and hence is a supervised technique.
Hence, when one discusses using dimensionality reduction not for visualization purposes (which would require a retention of shape) but to increase model performance, usually they are talking about LDA. Both, however, represent the newly formed dimensions in terms of linear combinations of dimensions in the dataset. Beyond simply support for model-building, however, it has proven itself a powerful method for analysis and interpretation.
Performing Linear Discriminant Analysis is a three-step process.
I. Calculate the ‘separability’ between the classes. Known as the between-class variance, it is defined as the distance between the mean of different classes, and allows for the algorithm to put a quantitative measure on ‘how difficult’ the problem is (closer…
