PyTorch Zero to Hero (Overview)~ 1

Let’s dive deeper into the higher dimensions of the tensors, revisit the concepts of PyTorch and gain basic knowledge to make our foundation strong.

Source

But what are tensors???

Tensors are a kind of specialized data structure that resembles arrays and matrices. Tensors are multi-dimensional arrays of numerical values with uniform types [1].

The rank (or order) of a tensor is determined by the number of dimensions (or directions) needed to describe it [2].

Let’s create some basic tensors

• Scalar Tensors: It is simply a single number with a value and has “zero axes” and “rank-0”

import torch

# Creating a scalar with a single value
scalar = torch.tensor(5)  # 5 is the value of the scalar

# You can also create a scalar with the `torch.scalar_tensor` function
scalar = torch.scalar_tensor(5)

# Checking the type and shape
print("Type of scalar:", type(scalar))
print("Shape of scalar:", scalar.shape)
print("Rank of scalar:", scalar.dim()) 

## Output

Type of scalar: <class 'torch.Tensor'>
Shape of scalar: torch.Size([])
Rank of scalar: 0

• Vector Tensors: It is a one-dimensional array of numbers and has “one axis”. and “rank-1 ”

# Creating a vector with a list of values
vector = torch.tensor([1, 2, 3])

# Checking the type and shape
print("Type of vector:", type(vector))
print("Shape of vector:", vector.shape)
print("Rank of vector:", vector.dim()) 

## Output

Type of vector: <class 'torch.Tensor'>
Shape of vector: torch.Size([3])
Rank of vector: 1

• Matrix Tensors: It refers to a two-dimensional array of numbers having “two axes” and "rank-2.”

matrix = torch.tensor([[1, 2], [3, 4]])  

# Checking the type and shape
print("Type of matrix:", type(matrix))
print("Shape of matrix:", matrix.shape)
print("Rank of matrix:", matrix.dim()) 

## Output

Type of matrix: <class 'torch.Tensor'>
Shape of matrix: torch.Size([2, 2])
Rank of matrix: 2

Tensor in general may have more axes, it can be 3,4 or any greater number

tensor_3d = torch.tensor(([
  [[0, 1, 2, 3, 4],
   [5, 6, 7, 8, 9]],
  [[10, 11, 12, 13, 14],
   [15, 16, 17, 18, 19]],
  [[20, 21, 22, 23, 24],
   [25, 26, 27, 28, 29]],]))

# Checking the type and shape
print("Type of tensor_3d:", type(tensor_3d))
print("Shape of tensor_3d:", tensor_3d.shape)
print("Rank of tensor_3d:", tensor_3d.dim()) 

## Output

Type of tensor_3d: <class 'torch.Tensor'>
Shape of tensor_3d: torch.Size([3, 2, 5])
Rank of tensor_3d: 3

There are several methods to visualize a tensor with more than two axes.

Source

Tensor Initializion

Tensors can be initialized in multiple ways, such as

1. Directly from data

2. From NumPy array

3. From another existing tensor

4. Using random values

5. Using constant values

Tensor Attributes

Each torch.Tensor has a torch.dtype, torch.device

PyTorch has twelve different data types:

32-bit floating point, 64-bit floating point, 64-bit complex, 128-bit complex, 16-bit floating point 1, 16-bit floating point 2, 8-bit integer (unsigned), 8-bit integer (signed), 16-bit integer (signed), 32-bit integer (signed), 64-bit integer (signed), Boolean.

torch.device represents the device on which a torch.Tensor is or will be allocated. Those device types include “cpu” ,“cuda”, “mps”, “xpu”, “xla” or “meta”.

Source

Note: By default each tensor created in CPU and needs to be explicitly moved to GPU using.to method. Make sure to check for the GPU availabiltiy to allocate the tensor to GPU

As shown in the above image, if a CUDA device is not available, the tensor will be created on the CPU by default. However, if any supported device is available, such as MPS in this case, the tensor will be allocated to that device.

Operations on Tensors

There are multiple tensor operations including generators, random sampling, arithmetics, transformations, parallelism, etc. Will look into a few of those now and reserve others for upcoming blogs to cover it in great detail.

Arithmetic operations

“ * ” and “.mul” computes the element-wise product

“.matmul” computes matric multiplication

Slicing and Indexing:

Joining tensors

In-place operations:

For in_place operations, results are stored in an operand and are denoted by a _ suffix. For. e.g. x.add_(y), x.t_()

In-place operations save some memory, but can be problematic when computing derivatives because of an immediate loss of history. Hence, their use is discouraged. ~ Source

Tensor connection with NumPy

Tensors on the CPU and NumPy arrays can share the same memory, so modifying one will directly affect the other.

Tensor to NumPy array:

A change in the tensor is reflected in the NumPy array.

NumPy array to Tensor:

A change in the NumPy array is reflected in the tensor.

References:

Tensors - PyTorch Tutorials 2.3.0+cu121 documentation

WTF is a Tensor?!? - KDnuggets

What is a Tensor?

Introduction to Tensors | TensorFlow Core