Dynamics of Regularized regression part2(Machine Learning 2024)
Linear Discriminant Regularized Regression
Authors: Xin Bing, Bingqing Li, Marten Wegkamp
Abstract: Linear Discriminant Analysis (LDA) is an important classification approach. Its simple linear form makes it easy to interpret and it is capable to handle multi-class responses. It is closely related to other classical multivariate statistical techniques, such as Fisher’s discriminant analysis, canonical correlation analysis and linear regression. In this paper we strengthen its connection to multivariate response regression by characterizing the explicit relationship between the discriminant directions and the regression coefficient matrix. This key characterization leads to a new regression-based multi-class classification procedure that is flexible enough to deploy any existing structured, regularized, and even non-parametric, regression methods. Moreover, our new formulation is generically easy to analyze compared to existing regression-based LDA procedures. In particular, we provide complete theoretical guarantees for using the widely used ℓ1-regularization that has not yet been fully analyzed in the LDA context. Our theoretical findings are corroborated by extensive simulation studies and real data analysis.
