“I do address Over-fitting problem” says Lasso and Ridge Regression.
In this post ,I will tell you about “Why and when we use Lasso and Ridge Regression and their key difference.“
Suppose ,we have a best fit line where for one unit change of X ,Y is making a big change (Case of Steep Slope).In this case model can Over-fit the data .To counter this problem we have Lasso and Ridge Regression .Both are the regularization techniques .
1)- Lasso uses L1-regularization technique
2)-Ridge uses L2-regularization technique
Difference between Lasso and Ridge Regression lies in the penalty of the Cost function.
If you remember in Linear regression we try to minimize the cost function so as to achieve the best fit line. In these regularization techniques ,we further add penalty term with the cost function. And these penalty terms that makes all the difference.
The Key idea behind using both these is to penalize the features having higher slope.
In the above plot if we choose line1 as best fit line ,we will face over-fitting problem so to overcome this we will select dotted line as our best fit line since it will produce more generalized model(less bias, less variance).
Ridge Regression Cost function
Ridge adds “squared magnitude” of slope in the cost function to penalize the steeper slopes and thus reducing overfitting issue.
Lasso Regression Cost Function
Lasso Regression adds “absolute value of slope” to the cost function as penalty term .In addition to resolve Overfitting issue ,lasso also helps us in feature selection by removing the features having slope very less or near to zero i.e features having less importance.(keep in mind slope will not be exactly zero).
Lasso works very well with large number of features.
Note: lambda used in penalty term is a hyperparameter which can be found out using Cross-Validation .
If you really find it insightful, don’t forget to clap
Happy Learning.
