PyTorch Tensors — quick reference

Hello!

This blog is part of the Torch Thursdays series. The plan is to share some tidbits on PyTorch usage through this. Today’s blog particularly is to share some quick notes based on PyTorch’s video tutorial on tensors. This blog assumes familiarity with PyTorch framework and numpy.

Tensor

PyTorch provides torch.Tensor to represent a multi-dimensional array containing elements of a single data type.

It is basically the same as a numpy array: it does not know anything about deep learning or computational graphs or gradients, and is just a generic n-dimensional array to be used for arbitrary numeric computation. The biggest difference between a numpy array and a PyTorch Tensor is that a PyTorch Tensor can run on either CPU or GPU. To run operations on the GPU, just cast the Tensor to a cuda datatype.

By default, the array elements are stored contiguously in memory leading to efficient implementations of various array processing algorithms that relay on the fast access to array elements. (note — contiguous arrays may not be efficient with sparse arrays , read more).

torch.tensor attributes

Tensor Attributes - PyTorch 1.10.1 documentation

There are three attributes : torch.dtype, torch.device and torch.layout

Tensor type — torch.dtype

• Torch tensor is created with FP32 data type by default, use dtype argument to set other types as needed (ex: int8 etc).

Making copies of tensors

• when you copy a tensor using “ = “ assignment, it creates another reference pointing to same location under the hood (ex : if a is a torch.tensor and we perform an assignment like b = a, any update to a , will also reflect on b. Essentially, they reference to same memory allocation).
• instead use b = a.clone(), this ensures you have made a separate copy. However, note that torch.requires_grad setting is copied as is from source tensor.
• In case we do not wish to copy the requires_grad setting, we should use detach() on source tensor during copy, like : c = a.detach().clone()

Tensor GPU usage — using torch.device

• check if faster hardware (GPU) is available : torch.cuda.is_available()
• create tensor on GPU device using device argument: a = torch.rand(2,2, device=”cuda”)
• moving data to device of interest :

a = torch.rand(2,2, device='cpu')
# transfer tensor created on cpu to gpu accessible memory.
if torch.cuda.is_available():
      device = torch.device('cuda') # create a device handle
       a = a.to(device) # pass device handle created.
# using string
torch.device('cuda')
torch.device('cpu')
torch.device('cuda:0')
# using string and device ordinal
torch.device('cuda', 0)
torch.device('cpu', 0)

Changing dimensions of tensors

• example when you want to change dimensions : you want to work on “batch” of input passed to model rather than single instance of input.
• use squeeze() and unsqueeze() (works on only dimension of extent 1)
• they can also be done in-place, using squeeze_() and unsqueeze_()

(did you know? — PyTorch methods with names ending with an underscore indicate they perform in-place operations).

# ex : image classification model
# image : 3,326,326 tensor
#  n * 326 * 326
a = torch.rand(3, 326, 326)
b = a.unsqueeze(0) # adds one extra dimension of extent 1.
# essentially extending dimension using unsqueeze retains the same number of elements as the original tensor. <becomes 1 x 3 x326 x 326>
c = b.squeeze(0) # back to (3,326,326)
#  ex : create data for batch of 1 during inference.

• reshape
• conv → linear (FC layer) in image classifier
• features * h * d → 1-D vector of FC layer — use reshape() method.

a3d = torch.rand(6, 20, 20)
a1d = a3d.reshape(6*20*20)
// note : reshape puts a view on orig tensor (uses same underlying memory of original tensor, no copy is made)

numpy bridge

• switching between numpy ndarrays and PyTorch tensor :

import numpy as np
# convert from numpy to PyTorch
np_array = np.ones((2,3))
pytorch_tensor = torch.from_numpy(np_array)
# convert from PyTorch to numpy
pt_tensor = torch.rand(2,3)
numpy_array = pt_tensor.numpy()
# they share same underlying memory. So, changes to one shall be reflected on the other. 
# for ex : pt_tensor update results in an update to numpy_array.

As always, if you found this blog useful, don’t forget to drop some claps (did you know? — you can add more than one clap :)) — so it can reach more folks. Thank you! Also, don’t forget to follow @howsofcoding and come along~

References

• More about torch.Tensor

PyTorch

NumPy