Sigmoid Function with PyTorch
In this article, I will tell you how to calculate the sigmoid(activation) function using PyTorch.
Sigmoid Function
First of all, we need to know what is the Sigmoid Function. Sigmoid Function is very commonly used in classifier algorithms to calculate the probability. It always returns a value between 0 and 1 which is the probability of a thing.
Read more about the sigmoid function in detail here.
PyTorch
PyTorch is a deep learning framework by the Facebook AI team. All deep learning frameworks have a backbone known as Tensor. You can think of tensor as a matrix or a vector i.e 1d tensor is a matrix, 2d tensor is the matrix, 3d tensors are array with 3 indices i.e RGB color code in 3d tensor. We can have n dimensions of the tensor. Let’s take a look at how we will calculate Activation(sigmoid function with PyTorch).
PyTorch tensors can be added, multiplied, subtracted, etc, just like Numpy arrays. In general, you’ll use PyTorch tensors pretty much the same way you’d use Numpy arrays.
Let’s Code it and Understand
Let's generate some random data using built-in methods of PyTorch.
We are using torch.manual.seed(), torch.randn(), torch.randn_like() to generate random data.
import torch # we import torch(i.e pyTorch)#Lets Generate some Random Datatorch.manual.seed(7) #set the seed
features = torch.randn((1,5))
#torch.randn takes tuple of dimensions and returns tensor of
#that dimension, here 1x5 vectorweights = torch.randn_like(features)
#randn_like takes a tensor and return a random tensor of that sizebias = torch.randn((1,1))
Now that we have generated random data lets code our sigmoid function
def sigmoid(x):
"""Sigmoid Activation Function
Arguments:
x.torch.tensor
Returns
Sigmoid(x.torch.tensor)
"""
return 1 / (1+torch.exp(x))
#remember formula of sigmoid
#function, torch.exp returns tensor of exp of all values in it>>> print(weights)
tensor([[-0.8948, -0.3556, 1.2324, 0.1382, -1.6822]])
>>>print(features)
tensor([[-0.1468, 0.7861, 0.9468, -1.1143, 1.6908]])
>>>print(bias)
tensor([[0.3177]])
1st Method
We calculate sigmoid function by sigmoid(w1x1+w2x2+….+wnxn+b) i.e we multiply weights and features which is the element by element multiplication i.e w1x1 + w2x2 +….+wnxn and add bias in the result. We send the whole result to the activation function and the answer is stored in y.
y = activation(torch.sum(features * weight)+bias)>>>print(y)
tensor([[0.1595]])
2nd Method
2nd Method is the Matrix Multiplication method. We multiply our features vector with weights vector using torch.matmul() or torch.mm() method which simply means torch.matrixmultiplication.
We use torch.mm() or torch.matmul() commonly in PyTorch. Here we simply multiply 2 vectors and add bias into it. Then we send it to the activation function and get the result. This method has a trick which we will see and that will help us a lot when we are going to deal with complex neural networks later. Let’s code it and see things. We are going to use previously declared variables and functions.
>>> print(weights)
tensor([[-0.8948, -0.3556, 1.2324, 0.1382, -1.6822]])
>>>print(features)
tensor([[-0.1468, 0.7861, 0.9468, -1.1143, 1.6908]])
>>>print(bias)
tensor([[0.3177]])y = activation(torch.mm(features , weight) + bias)
print(y)
It seems pretty simple, right? But here is a real problem. Both of our vectors are 5 x 1 dimension which means they can’t multiply i.e To multiply vectors/matrices, No. of columns of 1st must be equal to No. of Rows of 2nd which in our case is not happening. It returns the following error.
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-13-15d592eb5279> in <module>()
----> y=activation(torch.mm(features, weights)+ bias))RuntimeError: size mismatch, m1: [1 x 5], m2: [1 x 5] at /Users/soumith/minicondabuild3/conda-bld/pytorch_1524590658547/work/aten/src/TH/generic/THTensorMath.c:2033
This is a very common error and one of the most important ones to deal with. We have to reshape our tensors so that they can multiply. We have to reshape our weight vector to get the correct dimensions. Luckily we have some functions to deal with. We can use weights.reshape() or weights.resize() or weights.view(). There is a slight difference b/w them which you can read in the official documentation. We will use weights.view()
>>>weights.view(5,1)
tensor([[-0.8948],
[-0.3556],
[ 1.2324],
[ 0.1382],
[-1.6822]])Lets code it now
y = activation(torch.mm(features , weights.view(5,1))+ bias)>>>print(y)
tensor([[0.1595]])
You see that’s how we calculate the Activation function in PyTorch. It is even more powerful when we have multiple layers i.e hidden layers in our neural network.
Want to know more about Deep Learning with PyTorch?
Completely new to the field? Start here.
