Regression analysis with PyTorch

image courtesy

This is probably the thousandth article that is going to talk about implementing regression analysis using PyTorch. so how is it different?

Well, before I answer that let me write the series of events that led to this article. So, when I started learning PyTorch, I was excited but I had so many whys and why nots that I was frustrated at one point so, I thought why not start from scratch, understand the deep learning framework a little better and then delve deep into the complex concepts like CNN, RNN, LSTM etc. and the easiest way to do so is take a familiar dataset and explore as much as you can so that you understand the basic building blocks and the key working principle.

I have tried to explain the modules that are imported, why certain steps are mandatory and how we evaluate a model.

So people, if you have just started or looking for answers, like I did, then you are definitely in the right place. :)

okay, so let’s start with the imports first :

Imports

import torch
import torch.nn as nn

the torch.nn modules help us create and train neural networks. So definitely we need that. Let’s move on:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import MinMaxScaler

they look familiar, right? we need them cause we have to do some preprocessing on the dataset we will be using.

Data

The dataset I am going to use is celsius to fahrenheit data which can be found here: link

Data preprocessing step 1: separate out the feature and the label

X_train = train_data.iloc[:,0].values
y_train = train_data.iloc[:,-1].values

Data preprocessing step 2: standardize the data as the values are very large and varied. So if you don’t do that for this particular case, then later while training the model, you will likely get inf or nan for loss values, meaning, the model cannot perform back propagation properly and will result in a faulty model.

sc = MinMaxScaler()
sct = MinMaxScaler()
X_train=sc.fit_transform(X_train.reshape(-1,1))
y_train =sct.fit_transform(y_train.reshape(-1,1))

we have to make sure that X_train and y_train are 2-d.

Okay, so far so good. Now let’s enter into the world of tensors.

Data preprocessing step 3: Convert the numpy arrays to tensors

X_train = torch.from_numpy(X_train.astype(np.float32)).view(-1,1)
y_train = torch.from_numpy(y_train.astype(np.float32)).view(-1,1)

view takes care of the 2d thing in tensor as reshape does in numpy.

Model building

input_size = 1
output_size = 1

input= celsius

output = fahrenheit

Define layer

class LinearRegressionModel(torch.nn.Module):
    def __init__(self):
        super(LinearRegressionModel, self).__init__()
        self.linear = torch.nn.Linear(1, 1)  # One in and one out
    def forward(self, x):
        y_pred = self.linear(x)
        return y_pred

or we could have simply done this(since it is just a single layer)

model = nn.Linear(input_size , output_size)

In both the cases, we are using nn.Linear to create our first linear layer, this basically does a linear transformation on the data, say for a straight line it will be as simple as: y = w*x, where y is the label and x, the feature. Of course, w is the weight. In our data, celsius and fahrenheit follows a linear relation, so we are happy with one layer but in some cases where, the relation is non linear, we add additional steps to take care of the non linearity, say for example add a sigmoid function.

Define loss and optimizer

learning_rate = 0.0001
l = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr =learning_rate )

as you can see, the loss function in this case is “mse” or “mean squared error”. Our goal will be to reduce the loss and that can be done using an optimizer, in this case, stochastic gradient descent. That SGD needs an initial model parameters or weights and a learning rate.

Okay, now let’s start the training.

Training

num_epochs = 100
for epoch in range(num_epochs):
    #forward feed
    y_pred = model(X_train.requires_grad_())
    #calculate the loss
    loss= l(y_pred, y_train)
    #backward propagation: calculate gradients
    loss.backward()
    #update the weights
    optimizer.step()
    #clear out the gradients from the last step loss.backward()
    optimizer.zero_grad()
    
    print('epoch {}, loss {}'.format(epoch, loss.item()))

forward feed: in this phase we are just calculating the y_pred by using some initial weights and the feature values.

loss phase: after the y_pred, we need to measure how much prediction error happened. We are using mse to measure that.

back propagation: in this phase gradients are calculated.

step: the weights are now updated.

zero_grad: finally, clear the gradients from the last step and make room for the new ones.

Evaluation

predicted = model(X_train).detach().numpy()

detach() is saying that we do not need to store gradients any more so detach that from the tensor. Now, let’s visualize the model quality with the first 100 data points.

plt.scatter(X_train.detach().numpy()[:100] , y_train.detach().numpy()[:100])
plt.plot(X_train.detach().numpy()[:100] , predicted[:100] , "red")
plt.xlabel("Celcius")
plt.ylabel("Farenhite")
plt.show()

Notice, how with the increase of the number of the epochs, the predictions are getting better. There are multiple other strategies to optimize the network, example changing the learning rate, weight initialization techniques and so on.

Lastly, try with a known celsius value and see if the model is able to predict the fahrenheit value correctly. The values are transformed, so make sure you do a sc.inverse_transform() and sct.inverse_transform() to get the actual values.

Thank you for reading this article. Please do leave a comment or share your feedback. :)

References:

https://pytorch.org/tutorials/beginner/nn_tutorial.html

Linear Regression using PyTorch — GeeksforGeeks

What is the class definition of nn.Linear in pytorch

https://towardsdatascience.com/logistic-regression-on-mnist-with-pytorch-b048327f8d19