# How to implement Gradient Descent in Python

## This is a tutorial to implement Gradient Descent Algorithm for a single neuron

Gradient Descent is the most important concept in Neural Networks. In this tutorial, I am going to show you how to implement it in Python. I hope this tutorial can help you to build a better understanding about how gradient descent works, and how it helps to improve model accuracy.

# Data Pre-Processing

We will try to build a single neuron network, which can predict the admissions of a graduate school. The data we will use is shared above in google drive. Let us first take a peek at the raw data:

The first 5 rows of data are shown below. The first column `admit`

indicates whether the student is getting admitted to the school or not, this will be the **target** for our model; the second column `gre`

and the third column `gpa`

are **numerical features** for the student; the fourth column `rank`

is a **categorical feature**.

## One-hot Encoding

We will apply one-hot encoding to the categorical feature to add dummy columns. code shown as below:

## Data Normalization

We will also need to normalize the numerical features, code shown as below:

Finally, we shuffle the data, and split the entire data set into training and testing sets:

# Gradient Descent

## Activation Function

We will use `sigmoid`

function as the activation function for this neuron, code in python will be as below:

## Loss Function

One of the commonly used loss function for neural networks is `cross-entropy loss function`

. Since this algorithm will be used to predict admission into graduate schools, it will be a binary classification problem. So we will use `binary cross-entropy loss function`

. The code is implemented as below:

## Update Step

Below is the code for training the neuron and updating the weights:

Now we train the network and check the performance of our algorightm:

The running result is:

`Train loss: 0.6613580375785758`

Train loss: 0.5782721399941655

Train loss: 0.5782705527073997

Train loss: 0.5782705525116583

Train loss: 0.5782705525116338

Train loss: 0.5782705525116338

Train loss: 0.5782705525116338

Train loss: 0.578270552511634 WARNING - Loss Increasing

Train loss: 0.5782705525116338

Train loss: 0.5782705525116338

Prediction accuracy: 0.725