How to Apply L1 and L2 Regularization Techniques to Keras Models
Neural Networks and Deep Learning Course: Part 20
Prerequisite: Regularization Methods for Neural Networks — Introduction
There are several types of regularization techniques for neural networks. Today, we’ll discuss L1 and L2 regularization techniques and their Keras implementation.
Both L1 and L2 regularization techniques fall under the category of weight/parameter regularization. This type of regularization keeps the weights of the neural network small (near zero) by adding a penalizing term to the loss function.
How L1 and L2 regularization work to mitigate overfitting in neural networks
As we already discussed in Part 1, the weights in a neural network determine the level of importance of each input feature on the final output. Large weights put too much emphasis on some inputs that may be important when training the models, but less important when testing the model on new unseen data. That will result in high variance and overfitting in the model.