Binary Classifier using PyTorch

A simple binary classifier using PyTorch on scikit learn dataset

In this post I’m going to implement a simple binary classifier using PyTorch library and train it on a sample dataset generated using sklearn. I’ve tried searching for implementations on google, but couldn’t find any simple examples. Most of the examples I found were on image classification. The Goal of this post is not to teach about Classifiers or PyTorch. There are tons of resources floating on the web for that. I’ll just walk you through the implementation of the classifier.

Let’s generate dataset first

I’m generating 200 samples

import sklearn.datasets
X,y = sklearn.datasets.make_moons(200,noise=0.2) 

Let’s display the generated data using scatter plot

import matplotlib.pyplot as plt
plt.scatter(X[:,0],X[:,1],s=40,c=y,cmap=sklearn.cm.Spectral)

2 class data

We’ve generated a 2 class data. Let’s call the classes 0 and 1

We are going to build a classifier on the above data

Architecture of neural network we are gonna build looks like this

Architecture of neural network

It contains one input layer, one hidden layer and one output layer. Activation function I’m using for the hidden layer is tanh. Hidden layer contains 3 neurons. We can go for higher number of neurons or more number of hidden layers. But keep in mind that more neurons means, more parameters to train. Our data should be large enough then. So far, we have 6 + 6 = 12 weights from edges and 3 + 2 = 5 bias parameters. Total 17 trainable parameters.

The data that we’ve generated is of numpy datatype. We need to convert X and y to tensors (PyTorch datatype) to build a model. Let’s do that

#from_numpy takes a numpy element and returns torch.tensor
X = torch.from_numpy(X).type(torch.FloatTensor)
y = torch.from_numpy(y).type(torch.LongTensor)

Now X and y are tensors

Let’s build our neural network by defining a class

import torch.nn as nn
import torch.nn.functional as F
#our class must extend nn.Module
class MyClassifier(nn.Module):
    def __init__(self):
        super(MyClassifier,self).__init__()
        #Our network consists of 3 layers. 1 input, 1 hidden and 1 output layer
        #This applies Linear transformation to input data. 
        self.fc1 = nn.Linear(2,3)
        
        #This applies linear transformation to produce output data
        self.fc2 = nn.Linear(3,2)
        
    #This must be implemented
    def forward(self,x):
        #Output of the first layer
        x = self.fc1(x)
        #Activation function is Relu. Feel free to experiment with this
        x = F.tanh(x)
        #This produces output
        x = self.fc2(x)
        return x
        
    #This function takes an input and predicts the class, (0 or 1)        
    def predict(self,x):
        #Apply softmax to output. 
        pred = F.softmax(self.forward(x))
        ans = []
        #Pick the class with maximum weight
        for t in pred:
            if t[0]>t[1]:
                ans.append(0)
            else:
                ans.append(1)
        return torch.tensor(ans)

The criterion that we are going to use is CrossEntropyLoss. This is the common choice for most classification problems. Let’s train the model using ADAM(Adaptive moment estimation) method.

#Initialize the model        
model = Net()
#Define loss criterion
criterion = nn.CrossEntropyLoss()
#Define the optimizer
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

Let the training begin

#Number of epochs
epochs = 10000
#List to store losses
losses = []
for i in range(epochs):
    #Precit the output for Given input
    y_pred = model.forward(X)
    #Compute Cross entropy loss
    loss = criterion(y_pred,y)
    #Add loss to the list
    losses.append(loss.item())
    #Clear the previous gradients
    optimizer.zero_grad()
    #Compute gradients
    loss.backward()
    #Adjust weights
    optimizer.step()

I know this seems very confusing at first. All we are doing is, computing the loss. Calculating gradients and adjusting weights. implement couple of networks using PyTorch, you will get used to it for sure.

We are done with training process. Now, let’s calculate re-submission error. It’s the accuracy that we’ve been able to achieve on the training dataset. This is a bad practice, as we should always test the performance of a model on unseen data. Never the data that we have trained the model with.

from sklearn.metrics import accuracy_score
print(accuracy_score(model.predict(X),y))
OUTPUT : 
0.97

Let’s visualize the performance of the model by visualizing the decision boundary

This is a function which returns the class (0 or 1) , given a single input

def predict(x):
 #Convert into numpy element to tensor
 x = torch.from_numpy(x).type(torch.FloatTensor)
 #Predict and return ans
 ans = model.predict(x)
 return ans.numpy()

Below function plots the decision boundary. This is a very useful function , it gives a visual representation of how well your model is trained.

# Helper function to plot a decision boundary.
def plot_decision_boundary(pred_func,X,y):
    # Set min and max values and give it some padding
    x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5
    y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5
    h = 0.01
    # Generate a grid of points with distance h between them
    xx,yy=np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
    # Predict the function value for the whole gid
    Z = pred_func(np.c_[xx.ravel(), yy.ravel()])
    Z = Z.reshape(xx.shape)
    # Plot the contour and training examples
    plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral)
    plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.binary)

Finally , invoke plot_decision_boundary function to visualize the boundary

plot_decision_boundary(lambda x : predict(x) ,X.numpy(), y.numpy())

Decision Boundary:

I’d say, this classifier did a fairly good job. What do you think?

You can find the entire working code at

https://github.com/prudvinit/MyML/blob/master/lib/neural%20networks/pytorch%20moons.py