How To Use Regularization in Machine Learning?
Have you ever encountered a situation where your machine learning model models the training data exceptionally well but fails to perform well on the testing data i.e. was not able to predict test data? This situation can be dealt with with regularization in Machine learning.
Overfitting happens when a model learns the very specific pattern and noise from the training data to such an extent that it negatively impacts our model’s ability to generalize from our training data to new (“unseen”) data. By noise, we mean the irrelevant information or randomness in a dataset.
Preventing overfitting is very necessary to improve the performance of our machine learning model.
Following pointers will be covered in this article and eventually help us solve this problem,
- What is Regularization
- How Does Regularization Work
- Regularization Techniques
Moving on with this article on Regularization in Machine Learning.
What is Regularization?
In general, regularization means to make things regular or acceptable. This is exactly why we use it for applied machine learning. In the context of machine learning, regularization is the process that regularizes or shrinks the coefficients towards zero. In simple words, regularization discourages learning a more complex or flexible model, to prevent overfitting.
Moving on with this article on Regularization in Machine Learning.
How Does Regularization Work?
The basic idea is to penalize the complex models i.e. adding a complexity term that would give a bigger loss for complex models. To understand it, let’s consider a simple relation for linear regression. Mathematically, it is stated as below:
Y≈ W_0+ W_1 X_1+ W_2 X_(2 )+⋯+W_P X_P
Where Y is the learned relation i.e. the value to be predicted.
X_1,X_(2 ),〖…, X〗_P , are the features deciding the value of Y.
W_1,W_(2 ),〖…, W〗_P , are the weights attached to the features X_1, X_(2 ),〖…,X〗_P respectively.
W_0 represents the bias.
Now, in order to fit a model that accurately predicts the value of Y, we require a loss function and optimized parameters i.e. bias and weights.
The loss function generally used for linear regression is called the residual sum of squares (RSS). According to the above stated linear regression relation, it can be given as:
RSS= ∑_(j=1)^m (Y_i-W_0-∑_(i=1)^n W_i X_ji )²
We can also call RSS the linear regression objective without regularization.
Now, the model will learn by the means of this loss function. Based on our training data, it will adjust the weights (coefficients). If our dataset is noisy, it will face overfitting problems and estimated coefficients won’t generalize on the unseen data.
This is where regularization comes into action. It regularizes these learned estimates towards zero by penalizing the magnitude of coefficients.
But how it assigns a penalty to the coefficients, let’s explore.
Moving on with this article on Regularization in Machine Learning.
Regularization Techniques
There are two main regularization techniques, namely Ridge Regression and Lasso Regression. They both differ in the way they assign a penalty to the coefficients.
Ridge Regression (L2 Regularization)
This regularization technique performs L2 regularization. It modifies the RSS by adding the penalty (shrinkage quantity) equivalent to the square of the magnitude of coefficients.
∑_(j=1)^m (Y_i-W_0-∑_(i=1)^n W_i X_ji )²+ α∑_(i=1)^n W_i²=RSS+ α∑_(i=1)^n W_i²
Now, the coefficients are estimated using this modified loss function.
In the above equation, you may have noticed the parameter α (alpha) along with shrinkage quantity. This is called a tuning parameter that decides how much we want to penalize our model. In other terms, the tuning parameter balances the amount of emphasis given to minimizing RSS vs minimizing the sum of the square of coefficients.
Let’s see how the value of α alpha affects the estimates produced by ridge regression.
When α=0, the penalty term has no effect. It means it returns the residual sum of the square as the loss function which we choose initially i.e. we will get the same coefficients as simple linear regression.
When α=∞, the ridge regression coefficient will be zero because the modified loss function will ignore the core loss function and minimize coefficients square and eventually end up taking the parameter’s value as 0.
When 0<α<∞, for simple linear regression, the ridge regression coefficient will be somewhere between 0 and 1.
That’s the reason for selecting a good value of α (alpha) is critical. The coefficient methods produced by the ridge regression regularization technique are also known as the L2 norm.
Lasso Regression (L1 Regularization)
This regularization technique performs L1 regularization. It modifies the RSS by adding the penalty (shrinkage quantity) equivalent to the sum of the absolute value of coefficients.
∑_(j=1)^m (Y_i-W_0-∑_(i=1)^n W_i X_ji )²+ α∑_(i=1)^n |W_i |=RSS+ α∑_(i=1)^n |W_i |
Now, the coefficients are estimated using this modified loss function.
Lasso Regression is different from ridge regression as it uses absolute coefficient values for normalization.
As the loss function only considers absolute coefficients (weights), the optimization algorithm will penalize high coefficients. This is known as the L1 norm.
Here, α (alpha) is again a tuning parameter, works like that of ridge regression, and provides a tradeoff between balancing RS magnitude of coefficients.
Like ridge regression, α (alpha) in lasso regression can take various values as follows:
When α=0, we will get the same coefficients as simple linear regression.
When α=∞, the lasso regression coefficient will be zero.
When 0<α<∞, for simple linear regression, the lasso regression coefficient will be somewhere between 0 and 1.
It is appearing very similar to ridge regression, but let’s have a look at both techniques with a different perspective.
Think of ridge regression as solving an equation, where the sum of squares of weights(coefficients) is less than or equal to s. According to this, considering there are 2 parameters in a given problem, the ridge regression is expressed by
W_1²+ W_2²≤s
It implies that ridge regression coefficients have the smallest loss function for all points that lie within the circle given by the above equation.
Similarly, think of lasso regression as solving an equation, where the sum of modulus of weights(coefficients) is less than or equal to s. According to this, considering there are 2 parameters in a given problem, the lasso regression is expressed by
|W_1 |+ |W_2 |≤s
It implies that ridge regression coefficients have the smallest loss function for all points that lie within the diamond given by the above equation.
The following image describes the above equations:
In this image we can see, Constraint functions (blue area); the left one is for lasso whereas the right one is for the ridge, along with contours (green eclipse) for loss function i.e. RSS.
In the above case, for both regression techniques, the coefficient estimates are given by the first point at which contours (an eclipse) contacts the constraint (circle or diamond) region.
The ridge regression coefficient estimates will be exclusively non-zero. Why? Because ridge regression has a circular constraint, having no sharp points, the eclipse will not intersect the constraint on an axis.
On the other hand, the lasso constraint, because of its diamond shape, has corners at each of the axes hence the eclipse will often intersect at each of the axes. Due to that, at least one of the coefficients will equal zero.
The above scenario shows that ridge regression will shrink the coefficients very close to 0 but will never make them exactly 0, which means the final model will include all predictors. This is a disadvantage of ridge regression, called model interpretability.
However, lasso regression, when α is sufficiently large, will shrink some of the coefficient estimates to exactly 0. That’s the reason lasso provides sparse solutions.
That was Regularization and its techniques, and I hope now you can comprehend it in a better way. You can use this to improve the accuracy of your machine learning models.
Now, with this, we come to the end of this ‘Regularization in Machine Learning’ article. Hope this article was insightful!
If you have any queries regarding this topic, please leave a comment below and we’ll get back to you. If you wish to check out more articles on the market’s most trending technologies like Python, DevOps, Ethical Hacking, then you can refer to Edureka’s official site.
Do look out for other articles in this series that will explain the various other aspects of Data Science.
2.Math And Statistics For Data Science
9.Introduction To Machine Learning
12.How To Create A Perfect Decision Tree?
13.Top 10 Myths Regarding Data Scientists Roles
14.Top 5 Machine Learning Algorithms
15.Data Analyst vs Data Engineer vs Data Scientist
16.Types Of Artificial Intelligence
17.R vs Python
18.Artificial Intelligence vs Machine Learning vs Deep Learning
20.Data Analyst Interview Questions And Answers
21.Data Science And Machine Learning Tools For Non-Programmers
22.Top 10 Machine Learning Frameworks
23.Statistics for Machine Learning
25.Breadth-First Search Algorithm
26.Linear Discriminant Analysis in R
27.Prerequisites for Machine Learning
28.Interactive WebApps using R Shiny
29.Top 10 Books for Machine Learning
Originally published at https://www.edureka.co on August 7, 2019.