Introduction of Linear Discriminant Analysis (LDA)
/ November 15, 2017

LDA is widely used to find linear combinations of features while preserving class separability. Unlike PCA, LDA tries to model the differences between classes. Classic LDA is designed to take into account only two classes. Specifically, it requires data points for different classes to be far from each other, while points from the same class are close. Consequently, LDA obtains differenced projection vectors for each class. Multi-class LDA algorithms which can manage more than two classes are more used. Linear Discriminant Analysis (LDA) is most commonly used as dimensionality reduction technique in the pre-processing step for pattern-classification and machine learning applications. The goal is to project a dataset onto a lower-dimensional space with good class-separability in order avoid over fitting (“curse of dimensionality”) and also reduce computational costs. Summarizing the LDA approach in 5 steps Listed below are the 5 general steps for performing a linear discriminant analysis; we will explore them in more detail in the following sections. Compute the d-dimensional mean vectors for the different classes from the dataset. Compute the scatter matrices (in-between-class and within-class scatter matrix). Compute the eigenvectors ​$$(e_1,e_2,…e_d)$$​ and corresponding eigenvalues ​$$(λ_1,λ_2,…λ_d)$$​ for the scatter matrices. Sort the eigenvectors by decreasing eigenvalues…

Insert math as
$${}$$