Stochastic Gradient Descent vs Gradient Descent

What is Stochastic Gradient Descent algorithm and how is it different from Gradient Descent algorithm?

✅ Stochastic Gradient Descent (SGD) is a variant of Gradient Descent (GD) algorithm. Both are optimization algorithms used to train machine learning and deep learning algorithms. Their common objective is to find the best value of parameters which minimizes the loss function.

✅ In basic Gradient Descent (also known as Batch Gradient Descent or Vanilla Gradient Descent), we calculate the loss by taking into account the error values of predictions from the entire dataset and adjust the parameters based on that. While in Stochastic Gradient Descent, we randomly choose a data point, calculate error based on the prediction made by that single data point and adjust the parameters based on that. This process is repeated multiple times for the entire dataset.

✅ Since in GD, the loss is calculated by taking into account the entire dataset, the loss function tends to have a smoother convergence as compared to SGD which may lead to oscillating convergence due to updates based on single point. But the noisy and oscillating behavior of SGD can help to escape local minima especially in non-convex problems.

✅ Since GD consider the entire dataset in each iteration, it can be computationally more expensive than SGD which uses only one data point at a time. So, SGD can be a better choice for large datasets.

Hope this helps!

Please feel free to add your comments related to the topic in the comments section.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

I like understanding Machine Learning/Deep Learning algorithms at conceptual level and sharing my knowledge with simple explanations/diagrams. I believe having a clear understanding of the basics helps in building a strong foundation in these emerging areas 🙂

If you are looking to read similar stuff on regular basis,

Follow Kavita Gupta, PhD