In our last post, we have discussed the difficulties of setting the learning rate hyper-parameter, which can be mitigated by using the Adagrad algorithm. However, this algorithm has a serious flaw, i.e., the effective learning rate…
We have discussed several algorithms in the last two posts, and there is a hyper-parameter that used in all algorithms, i.e., the learning rate (η). To refresh again, a hyper-parameter is a parameter that has to be chosen manually before training. The prefix ‘hyper-‘ is to…
In the previous post, Gradient Descent and Stochastic Gradient Descent Algorithms for Neural Networks, we have discussed how Stochastic Gradient Descent algorithm can improve plain Gradient Descent algorithm in terms of complexity and…
