All You Need to Know Of Optimization Algorithm in Deep Learning

It is known that deep learning algorithms involve optimization in many many contexts. In practice, we often use analytical optimization to design algorithms. However, given the complexity of deep learning, it is quite common to invest days or even months of time across hundreds of machines to solve just a few instances of neural network training. Since the problem is so important…