Regularization in Machine Learning

Regularization is a widespread technique in machine learning, which is used to control the complexity of the machine learning model and thereby improve its generalization ability.

In this article I’m going to explain the concept of regularization in depth, and demonstrate its usage in several machine learning models.

What is Regularization?

Regularization is a technique to prevent overfitting by penalizing complex models. The idea is to add a penalty term to the cost function of the model, such that it becomes dependent on two factors:

Cost(h) = Training Error(h) + λ Complexity(h)

λ is a hyperparameter (called the regularization coefficient) that controls the tradeoff between the bias and the variance (for a discussion on the bias-variance tradeoff see my previous article). Higher λ will induce a larger penalty on the complexity of the model, and thus will lead to simpler models with higher error on the training set but with smaller variance.

The complexity of the model can be measured in a variety of ways. For example, in models that consist of a vector of parameters (weights) w, such as linear regression or neural networks, we use the size of the parameters (the norm of the vector w) as a measure for the model’s complexity.

In such models, there are two common types of regularization, depending on the norm of the vector w that we are using: