Types of Optimization Algorithms used in Neural Networks and Ways to Optimize Gradient Descent
Anish Singh Walia
1.6K18

You might mention Newton-Raphson for a single variable and its generalization; also conjugate gradient methods. There is also the question of optimal step size choice in the gradient direction. I have n’t followed the literature in the last 25 years or so, though but things don’t appear to have changed too drastically. Your article is very well-written and enjoyable. Thanks

Like what you read? Give Raj Rajan a round of applause.

From a quick cheer to a standing ovation, clap to show how much you enjoyed this story.