How to implement Gradient Descent in Python

This is a tutorial to implement Gradient Descent Algorithm for a single neuron

Y Tech
Y Tech
Aug 14, 2019 · 2 min read
Image for post
Image for post

Gradient Descent is the most important concept in Neural Networks. In this tutorial, I am going to show you how to implement it in Python. I hope this tutorial can help you to build a better understanding about how gradient descent works, and how it helps to improve model accuracy.

Data Pre-Processing

We will try to build a single neuron network, which can predict the admissions of a graduate school. The data we will use is shared above in google drive. Let us first take a peek at the raw data:

The first 5 rows of data are shown below. The first column admit indicates whether the student is getting admitted to the school or not, this will be the target for our model; the second column gre and the third column gpa are numerical features for the student; the fourth column rank is a categorical feature.

Image for post
Image for post
First 5 Rows of Data

One-hot Encoding

We will apply one-hot encoding to the categorical feature to add dummy columns. code shown as below:

Data Normalization

We will also need to normalize the numerical features, code shown as below:

Finally, we shuffle the data, and split the entire data set into training and testing sets:

Gradient Descent

Activation Function

We will use sigmoid function as the activation function for this neuron, code in python will be as below:

Loss Function

One of the commonly used loss function for neural networks is cross-entropy loss function. Since this algorithm will be used to predict admission into graduate schools, it will be a binary classification problem. So we will use binary cross-entropy loss function. The code is implemented as below:

Update Step

Below is the code for training the neuron and updating the weights:

Now we train the network and check the performance of our algorightm:

The running result is:

Train loss:  0.6613580375785758
Train loss: 0.5782721399941655
Train loss: 0.5782705527073997
Train loss: 0.5782705525116583
Train loss: 0.5782705525116338
Train loss: 0.5782705525116338
Train loss: 0.5782705525116338
Train loss: 0.578270552511634 WARNING - Loss Increasing
Train loss: 0.5782705525116338
Train loss: 0.5782705525116338
Prediction accuracy: 0.725

The Startup

Medium's largest active publication, followed by +773K people. Follow to join our community.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store