Implementing a Logistic Regression Model from Scratch with PyTorch

Learn how to implement the fundamental building blocks of a neural network using PyTorch.

In this tutorial, we are going to implement a logistic regression model from scratch with PyTorch. The model will be designed with neural networks in mind and will be used for a simple image classification task. I believe this is a great approach to begin understanding the fundamental building blocks behind a neural network. Additionally, we will also look at best practices on how to use PyTorch for training neural networks.

After completing this tutorial the learner is expected to know the basic building blocks of a logistic regression model. The learner is also expected to apply the logistic regression model to a binary image classification problem of their choice using PyTorch code.

Check if GPU is available:

Importing Dataset

In this tutorial, we will be working on an image classification problem. You can find the public dataset here.

The objective of our model is to learn to classify “bee” vs. “no bee” images.

Since we are using Google Colab, we will need to first import our data into our environment using the code below:

Paste the code provided by the authorization page into the input box.

Note: If you are interested in jumping straight to the Colab/Jupyter notebook, please refer to the link provided at the end of this blog post.

Data Transformation

This is an image classification task, which means that we need to perform a few transformations on our dataset before we train our models. I used similar transformations as used in this tutorial. For a detailed overview of each transformation take a look at the official torchvision documentation.

The following code block performs the following operations:

• The data_transforms contains a series of transformations that will be performed on each image found in the dataset. This includes cropping the image, resizing the image, converting it to tensor, reshaping it, and normalizing it.
• Once those transformations have been defined, then the DataLoader function is used to automatically load the datasets and perform any additional configuration such as shuffling, batches, etc.

Print sample

It’s always a good practice taking a quick look at the dataset before training your models. Below we print out an example of one of the images from the train_dataset.

the output

Building the Model

Let’s now implement our logistic regression model. Logistic regression is one in a family of machine learning techniques that are used to train binary classifiers. They are also a great way to understand the fundamental building blocks of neural networks, thus they can also be considered the simplest of neural networks where the model performs a forward and abackward propagation to train the model on the data provided.

If you don’t fully understand the structure of the code below, I strongly recommend you to read the following tutorial, which I wrote for PyTorch beginners. You can also check out Week 2 of Andrew Ng’s Deep Learning Specialization course for all the explanation, math, intuitions, and details of the different parts of the neural network such as the forward, sigmoid, backward, and optimization steps.

In short:

• The __init__ function initializes all the parameters (W, b, grad) that will be used to train the model through backpropagation.
• The goal is to learn the W and b that minimizes the cost function which is computed as seen in the loss function below.

Note that this is a very detailed implementation of a logistic regression model so I had to explicitly move a lot of the computations into the GPU for faster calculation, .to(device) takes care of this in PyTorch. I decided to place all my variables to the GPU for computation but I can choose what goes where.

Pretesting the Model

It is also good practice to test your model and make sure the right steps are taking place before training the entire model.

Output:

Cost: tensor(0.6931)

Accuracy: tensor(50.4098)

Train the Model

It’s now time to train the model.

The output from the code above:

Result

From the results above and the learning curve below you can see that the model is sort of learning to classify the images given the decrease in the loss. I only ran the model for 100 iterations. You can try to train the model for many more rounds and analyze the results. In fact, I have suggested a couple of experiments and exercises at the end of the tutorial that you can try to improve the model.

The output from the code above:

Exercises 📚

There are many improvements and different experiments that you can perform on top of this notebook to keep improving your ML skills:

• It is always good to normalize/standardize your images which helps with learning. As an experiment, you can research and try different ways to standardize the dataset. We have normalized the dataset with the builtin PyTorch normalizer which uses the mean and standard deviation. Alternatively, you can simply divide the original pixel values by 255 which is what a lot of ML engineers do. Play around with this idea, and try different transformation or normalization techniques. What effect does this have on learning in terms of speed and loss?
• The dataset is too small so our model is not really learning effectively. You can try many things to help with learning such as playing around with the learning rate. Try to decrease and increase the learning rate and observe the effect of this on the model when training?
• If you explored the dataset further, you may have noticed that all the “no bee” images are actually “ant” images. If you would like to create a more robust model, you may want to make your “no bee” images more random and diverse. Additionally, the dataset is also being shuffled which you can easily disable in the data transformation section. What happens if you disable the shuffling?
• The model is not really performing that well because of the dataset I am using and because I didn’t train it for long enough. It is a relatively small dataset but the performance should get better with more training over time. A more challenging task involves adopting the model to other datasets. Give it a try and let me know about your results!
• Another important part that is missing in this tutorial is a comprehensive analysis of the model results. If you understood the code, it should be easy to figure out how to test with a few examples. In fact, it would also be great if you can put aside a small testing dataset for this part of the exercise, so as to test the generalization capabilities of the model.
• We built the logistic regression model from scratch but with libraries like PyTorch, these days you can simply leverage the high-level functions that implement certain parts of the neural network for you. This simplifies your code and minimizes the number of bugs in your code. Plus you don’t have to code your neural networks from scratch all the time. As a bonus exercise, try to adapt PyTorch builtin modules and functions for implementing a simpler version of the above logistic regression model. I will also add this as a to-do task for myself and post a solution soon.

References

• Understanding the Impact of Learning Rate on Neural Network Performance
• Transfer learning for computer vision tutorial
• Deep Learning Specialization by Andrew Ng

>> GitHub | Google Colab <<

If you spot any errors/bugs in the tutorial, please leave your comments below and I will address them as soon as I can. I appreciate it in advance. 😺