One particular layer that is useful, yet mysterious when training neural networks is Dropout. Dropout is created as a…
Our last post discussed two other adaptive algorithms which are an extension to the Adagrad algorithm, i.e., Adadelta and RMSProp. As we have discussed, we can view the accumulated gradients in Adadelta as the acceleration…
In our last post, we have discussed the difficulties of setting the learning rate hyper-parameter, which can be mitigated by using the Adagrad algorithm. However, this algorithm has a serious flaw, i.e., the effective learning rate…
We have discussed several algorithms in the last two posts, and there is a hyper-parameter that used in all algorithms, i.e., the learning rate (η). To refresh again, a hyper-parameter is a parameter that has to be chosen manually before training. The prefix ‘hyper-‘ is to…
In the previous post, Gradient Descent and Stochastic Gradient Descent Algorithms for Neural Networks, we have discussed how Stochastic Gradient Descent algorithm can improve plain Gradient Descent algorithm in terms of complexity and…
Everyone who ever have trained Neural Networks, chances are, have been stumbled with Gradient Descent algorithm or its variations. These algorithms are used to find parameter that minimize the…
