Mathematics

Gradient Descent Visualization

Visualize SGD optimization algorithm with Python & Jupyter

Introduction

In this article I am aiming to provide a good visual perspective to understand the Stochastic Gradient Descent (SGB for short) algorithm, this will hopefully give a good understanding to You.

Some of the math is represented in this article, but I advise You to read further on other materials for a deeper understanding.

Visualizations were made in Python, with the help of Jupyter Notebook, Matplotlib, Numpy libraries.

Have fun!

Use cases

SGD is mainly used in math and engineering sciences. I plan to approach SGD with a Computer Scientist (Machine Learning) mindset.

The most used case of SGD is in Machine Learning (or Deep Learning), where it is used to search for global function minimum. Namely, to search for the global minimum of the Cost function.

Note: The algorithm does not always find the global minimum but that is not in the scope of this article.

Gradient Descent

The simpliest case of SGD:

• We have a function C.