Member-only story
Learning Rate Schedules and Decay in Keras Optimizers
Options for changing the learning rate during training
Importance of dynamic learning rates
The learning rate is one of the most important neural network hyperparameters.
In a previous post, I discussed a systematic approach to choosing the optimal learning rate for neural networks. There, we focused on fixed learning rates.
When using a fixed learning rate, we change the learning rate value only after training!
The most effective way of using the learning rate is to decrease its value during training. This type of learning rate is known as a dynamic learning rate whose value is decreased over time!
A dynamic learning rate will allow the model to properly converge at the optimal point where the error is minimum. Sometimes, a static learning rate will miss that point even if with its best value! A dynamic learning rate will also reduce the time taken to converge for the model.
Main approaches to decrease the learning rate during training
There are mainly two approaches to decrease the learning rate during training.
- Using optimizers that have adaptive…
