What is Least Angle Regression (LAR)?
LAR and its algorithm.
LAR is a type of forward step-wise regression. LAR is connected to Lasso regression. LAR was defined in the Efron et al., 2004. It is a relatively newer algorithm and is viewed as a democratic version of the forward step-wise regression.
Forward step-wise regression:
Forward step-wise regression builds a model sequentially, adding one variable at a time. At each step, it identifies the best variable to include in the active set, and then updates the least squares fit to include all the active variables.
Least Angle Regression:
In LAR, the first step is to identify the variable most correlated with the response. Rather than fitting this variable completely, LAR moves the coefficient of this variable continuously towards its least squares value (causing its correlation with the evolving residual to decrease in absolute value). As soon as another variable ‘catches up’ in terms of correlation with the residual, the process is paused. The second variable then joins the active set, and their coefficients are moved together in a way that keeps their correlations tied and decreasing. In conclusion, Least angle regression only enters as much of a predictor as it deserves.
