What is a Tensor? Shape, Dims/Rank, and DType Explained
A Tensor is an array that carries numerical data. This array can have multiple dimensions that’s why people often define it as a multidimensional array.
Here data means anything that can be converted into its numerical form. For eg. let's take an image with a height & width of 640x640 that can be converted into a Tensor of the shape (you’ll learn about the shapes below) of [3, 640, 640] where 3 is the number of color channels which is 3 (Red, Green & Blue), and then comes the height 640 pixels and width 640pixels.
A Tensor is a basic building block in machine learning. Neural Networks operate on Tensors as their primary input and output data structure. So what goes into the Neural Networks is a Tensor and what comes out as a result is a Tensor.
A Tensor has some special characteristics known as Dimensions/Rank, Shape, and DType(Data Type) which makes it more special as an array.
Dimension/Rank
Dimensions or also called Rank in tensor-speak (language used for tensors) are simply the number of dimensions a tensor has. To know the dimension of a Tensor count it from one side let's say from the left and when you reach the non-array element/value you have got the number for the dimensions.
While a Tensor is just a special array it is also denoted with specific names at different ranks.
- Rank 0: A tensor with rank 0 is called a scalar. It represents a single value, such as a real number or a constant.
- Rank 1: A tensor with rank 1 is called a vector. Vectors have one-dimensional data and can be represented as a list of values.
- Rank 2: A tensor with rank 2 is called a matrix. Matrices have two dimensions and are used to represent 2D data structures, such as images or tabular data.
- Rank 3: A tensor with rank 3 is often referred to as a 3D tensor. It can be visualized as a cube or a stack of matrices.
- Rank 4: A tensor with rank 4 is often called a 4D tensor. In machine learning, 4D tensors are frequently used to represent images in batch form, where the fourth dimension represents the batch size.
Vice versa Tensors can have even higher ranks, depending on the problem and the data’s complexity. For example, in video data, you might encounter 5D tensors, where each element contains a 4D tensor representing a frame in the video.
Shape
The shape of a Tensor is related to its dimensions. It is an array of numbers representing the length of each dimension. This is the most important and interesting characteristic of a Tensor.
The way you identify the shape of a Tensor is you figure out the number of elements on each dimension.
In the above example, the Tensor has 3 dimensions and on each dimension, there are two elements that’s why the shape of the Tensor is [2, 2, 2]. It doesn’t matter if the elements are array or value both are counted as elements while figuring out the shape of the Tensor.
Most of the issues faced by machine learners are related to shape mismatching. You can’t pass a differently shaped Tensor into a Neural Network which was trained on a specific shape. You’ll have to prepare the Tensor with the exact shape the Network expects. That makes shape an important topic.
DType
The data type of a tensor defines the type of the values it can hold, such as integers, floating-point numbers, or booleans. Common data types include int32, float32, and bool. The data type of a tensor determines the precision and memory requirements for storing its elements.
Summary
Tensor is a special array that has some helpful characteristics that let us represent our data numerically. It’s the primary unit of data in machine learning frameworks like PyTorch, TensorFlow, and TensorFlow.js.
Neural Networks aren’t aware of strings, images, or audio. All they know is numbers represented in a low range and disciplined way that they will operate to find and learn the structures and patterns in the input Tensor and recognize the relationship with the output Tensor.
Thank you for Reading.
