Preventing overfitting: Regularization

The ultimate goal of any Machine Learning model is making reliable predictions on new, unknown data. Hence, while training our algorithm, we always have to keep in mind that having a good score in our train set doesn’t necessarily mean our model will adapt to new data well. Indeed, whenever we have a model which perfectly fit our training data, we are probably incurring in overfitting: our model has too many parameters which cannot be justified by data, hence it is way too complex and heavy.

Why does it happen? Well, if we think about the optimization strategy of any algorithm-minimizing the loss function-we can easily understand that, to reduce the error, our algorithm will tend to increase the value/number of our parameters in order to properly fit our data. However, this might lead to poor predictions on the test set.

To better understand this concept, let’s have a look at the following situation:

Here, red dots are the train set, while those blue are the test set. We can fit them with linear regression and, if our optimization strategy consists of reducing the loss function (let’s say, the Residual Sum of Errors loss function), we will have something like that: