Gradient Descent

Ahmed Imam
Apr 22 · 5 min read

Learn by example (Part — 1)

What is gradient descent?

Example:

Suppose that:

We have the following linear system

Image for post
Image for post
Equation 1.1

Where:

(𝑦̂) is predicted output

𝑠𝑙𝑜𝑝𝑒 = 𝛽1

𝑖𝑛𝑡𝑒𝑟𝑐𝑒𝑝𝑡 = 𝛽0

Image for post
Image for post
Equation 1.2

Where:

(𝑦) is actual output

Tip To get the idea of “Gradient Descent” we will work first on “intercept-coeff.” then on both of coefficients together (intercept & slope). So a fixed value for ‘slope’ will be used ( 𝛽1= 0.64)

Image for post
Image for post
Equation 1.3
Image for post
Image for post
Fig. 1: Relation between cost and intercept-coefficient.
Image for post
Image for post
Equation 1.4

Calculations:

Image for post
Image for post
Fig. 2 Slope at arbitrary point on cost-curve where (𝛽0=0)

2. Compute the step-size, gradient-descent will use to jump to next point on the curve.

Image for post
Image for post
Equation 1.5

3. Compute the new-𝛽0.

Image for post
Image for post

4. Repeat steps from “1” to “3” with the value of (new-𝛽0) again and again till the absolute value of step to be equal to or less than value of 0.001 (|step| ≤ 0.001) i.e. at this value the change in both step and new-𝛽0 will be too small.

5. At (|step| ≤ 0.001), gradient-descent approaches its goal to get the best value of system’s intercept-coeff. (𝛽0) with the minimum error (cost).

And hence our target is approached.

The following code is doing this:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
# Importing our dataset from CSV file:
dataset = pd.read_csv('./Datasets/weights.csv')
x = dataset['weight']
y = dataset['height']
# get relation between "intercept" coef. with squared error:
def sqr_err():
sqError = []
incpt = np.arange(0, 2, 0.1)
slope = 0.64
for i in incpt:
y_pred = i + slope * x
residual = sum((y_pred - y)**2)
sqError.append(residual) plt.plot(incpt, sqError)
plt.title('Intercept vs Squared_Error')
plt.xlabel('Intercept')
plt.ylabel('Squared Error')

The above part represents the cost curve relationship with 𝛽0.

# Draw Squared_Residual / Intercept curve:
sqr_err()
# Initial value of intercept
i = 0
learningrate = 0.1
while(True):
# Draw a certain value of Squared_Error to get slope at
# any point on cost-curve
y_pred = i + 0.64 * x
cost = sum((y_pred - y)**2)
plt.plot(i, cost, c='r', marker='o')

print(f'Intercept= {round(i,4)}')
print(f'Seqared-Residual: {round(cost, 4)}')

# Draw tangent
costSlope = -2*(1.4 - (i + 0.64 * 0.5)) - 2*(1.9 - (i + 0.64 * 2.3)) - 2*(3.2 - (i + 0.64 * 2.9))
print(f'Cost_Slope: {round(costSlope, 4)}')
line_x_values = np.arange(0, 1.2, 0.1)

# tangent will pass through (0, tangent_intrcept) and (i, cost) and have slope "costSlope"
tangent_intrcept = cost - (i * costSlope)
tangent = tangent_intrcept + costSlope * line_x_values
plt.plot(line_x_values, tangent)

stepSize = learningrate * costSlope
if abs(stepSize) <= 0.001:
print(f'Absolute value of Step Size is less than (0.001): {round(abs(stepSize), 4)}')
break

print(f'Step Size: {round(stepSize, 4)}')
print('\n')
# New Intercept = Old Intercept - Step Size
i = i - stepSize

plt.show()
Image for post
Image for post
Image for post
Image for post
Fig.3 How gradient descent works to get min. cost

The Second part of this tutorial will be about how to use gradient descent to get both of linear system coefficients (𝛽0 & 𝛽1) of equation-1.1 in the same approach our target which is min. cost.

Part-2:

https://medium.com/@ah.im.mo7/gradient-descent-5195ec6182bc

Analytics Vidhya

Analytics Vidhya is a community of Analytics and Data…

Ahmed Imam

Written by

Machine Learning Engineer & Python/Machine Learning Senior Instructor

Analytics Vidhya

Analytics Vidhya is a community of Analytics and Data Science professionals. We are building the next-gen data science ecosystem https://www.analyticsvidhya.com

Ahmed Imam

Written by

Machine Learning Engineer & Python/Machine Learning Senior Instructor

Analytics Vidhya

Analytics Vidhya is a community of Analytics and Data Science professionals. We are building the next-gen data science ecosystem https://www.analyticsvidhya.com

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store