Regularization | one minute introduction

This is a regular term in a lot of machine learning concepts

In a sense, pretty much all new machine learning models just try to find novel ways of performing regularization, especially faster or with less compute.

1. Why? If one just blindly trains a supervised deep learning model on a bunch of data for a long time, there is a good chance that the model will simply memorize the complex patterns of the training data and therefore not generalize well to new, “unseen” test data (i.e. the model will overfit)
2. What? Regularization techniques are those that try to discourage learning the specific patterns of the training data so as to prevent overfitting
3. How? There are many common techniques for regularization, such as performing random dropout of layers during training (dropout), penalizing larger weight values when calculating the loss (weight decay), or rescaling the inputs so that mini-batches have a certain desirable distribution (batch normalization)