Deep Dive into ML Algorithms -Logistic Regression

In this Article, will learn about logistic regression from scratch using python. I will explain stepwise the details of how logistic regression works and then we will code using the IRIS dataset.

Introduction

We will try to Learn logistic Regression from the neural network perspective. In simple terms logistic regression can be described as y = w*x +b where y is the dependent variable, x is the independent variable, w is weights of the equations or model and b is constant or bias of the model.

Unlike all regression methods, logistic regression is used for classification as it outputs(0 and 1) binary labels. In the linear regression method, we try to find the best trendline which fits the data well whereas in logistic regression we try to learn the best combinations of weights to classify the input correctly.

Implementation of Logistic Regression

Step 1: Data Set

the very first step is to get a dataset and visualize its distribution and distribution of classes. we will be using the IRIS data set and will use only two classes as we will try to implement binary label classification through logistic regression.

Importing Libraries

import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
import math
from numpy import *
import matplotlib.pyplot as plt
import random

Step 2: Decide your input and output labels

In our case, Input will be the dataset columns such as sepal length, sepal width, petal length (cm), petal width (cm). output labels will be classes of flowers as Setosa and Versicolor.

iris_data = load_iris(return_X_y=False)
df_iris = pd.DataFrame(data= np.c_[iris_data['data'], iris_data['target']],columns= iris_data['feature_names']+['target'])
# getting just 2 classes for binary classification
df_iris_sample = df_iris[df_iris["target"] != 2]
# Input features
X = df_iris_sample[['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)','petal width (cm)']].to_numpy()
# Output feature
y = df_iris_sample[['target']].to_numpy().ravel()

Step 3: prepare the test and train data

x_train,x_test,y_train,y_test = train_test_split(X,y,test_size=0.3)

Step 4: Logistic Regression begins ….

1. Convert Input into a NumPy matrix so it will be easy to process and NumPy also inherently applies multithreading concepts in processing.
2. encode your output to 0 and 1
3. In this equation z = w*x + b we are ready with x as input features and z is our output labels which we gonna use in training. what about w and b?
4. w (weights) and b(bias) are the parameters in logistic regression that needs to be learned while training the model.
5. Initially, we can assume random weights and bias using Numpy random module.
6. after filling weights and bias we will calculate the value of z in this equation (z = w*x + b )
7. after calculating the value of a linear equation we will apply a sigmoid activation function which can be seen in the figure. here z is the linear combination of weights and bias

8. why sigmoid? Our goal of binary classification requires that the output should be either 0 or 1 so sigmoid is a mathematical function that gives the probability-based results and restricts the output into 0 and 1.

9. Now we are done with calculating z and after applying sigmoid we have some output, In order to see whether this output is good or not we need to match it with true labels of data and applying some cost function(explain in detail in later sections) to see how good we have predicted.

10. After Computing the cost we realized, that we can do better by changing the weight and bias to get more accurate predictions.

11. how to change the weights and bias? the answer is gradient descent.

12. this process of calculating z and then sigmoid. compute the cost and then again do the gradient descent continues till you got the desired results or desired accuracy of the model.

Sigmoid Function

It is a type of activation function. In neural networks, we use various activation functions to alter or modify the output. Sigmoid also provides the same functionality as it outputs between 0 and 1 (probability). If the output of sigmoid is less than 0.5 we will classify that instance to 0 and if it's greater than 0.5, we will classify into 1.

Python Implementation

def sigmoid(z):
   return (1/(1+np.exp(-z)))
here z is calculated linear combination(z = w*x + b)

Cost Function

In order to see how well the model is performing at each iteration, we require a performance measure. In logistic Regression, there is a concept of cost or cross-entropy which basically sees how many input instances model has classified correctly. In the formula, we will take the mean because we want a number which can represent the cost of the whole sample or input(x)

Python Implementation

def cost_function(y,a):
   return -np.mean(y * np.log(a) + (1-y) * np.log(np.array(1) - a))
y : True labels
a : predicted labels

Gradient Descent

The most important concept of Logistic Regression is learning of weight and bias. To help in the learning process gradient descent helps us to know how much we need to update the weights and bias to get more accurate results.

Working of Gradient Descent

Initially, we start with random weights and bias and calculate the cost at each iteration which can be seen from the figure. at each iteration, we will try to update the weights in the direction of the slope of the cost curve so that we get the optimum value of weights. after achieving the optimum value of the weights, change in weights will be zero or very less.

You can start with any weights either on the positive side of the optimum value or negative side. however, while updating the weights you will converge in the direction of optimum weight.

Python Implementation

def Back_propogation(predicted,actual,X):
grad = predicted - actual
dw = np.dot(X.T,grad)/X.shape[0]
db = np.average(grad)
return dw,db

Gradient means slope and in each iteration, we calculate the differentiation of weights and bias to update the values of weight and bias for the next iterations. here alpha is the learning parameter. It controls the speed of learning. if alpha is too large then weights will be updated with high values and if its too less then weights will be updated slowly.

Step 5: Implement the logistic Regression

def logistic_regression_training(X, y, max_iter=1000, alpha=0.01, tol=1e-4):
random.seed(42)
w = []
cost = []
b = np.random.uniform(0,1)
d = 4
for j in range(0,d):
w.append(np.random.uniform(-0.01,0.01))
for i in range(max_iter):
   pred = sigmoid(np.dot(w,X.T) + b)
   dw,db = Back_propogation(pred,y,X)
   cost.append(cost_function(y,pred))
   w,b = gradient_descent(w,b,X,pred,y,alpha)
return w,b,cost

Step 6: Prediction from Logistic Regression

def logistic_regression_predict(x, W, b):
linear = np.dot(W,x.T) + B
y_pred = 1 / (1 + np.exp(-linear))
for i in range(len(y_pred)):
    if (y_pred[i] < 0.5):
         y_pred[i] = 0
    else:
         y_pred[i] = 1
return y_pred

Step 7: Run these function to get the weight and bias of the model

W,B,cost = logistic_regression_training(x_train,y_train)

Step 8: Do the predictions from weights and bias

y_pred = logistic_regression_predict(x_test,W,B)
accuracy = model_accuracy(y_test,y_pred)
print(accuracy)

Conclusion

In this article, we have learned about Logistic Regression and how can we implement logistic regression from scratch using python.

Happy Reading and coding !!