A Simple Introduction to Tensors
A tensor is a generalization of vectors and matrices to n dimensions. Understanding how they interact with each other is fundamental to machine learning.
Overview
Although tensors appear to be complex objects, they can be understood as a collection of vectors and matrices. Understanding vectors and matrices is essential to understanding tensors.
A vector is a one-dimensional list of elements:
A matrix is a two-dimensional list of vectors:
The subscript indicates the (row, column). Another way to think about a matrix is a vector with vectors as its elements. Note that they are commonly notated by capital letters.
A 3D tensor can be thought of as a three-dimensional list of matrices:
Another way to think about a 3D tensor is a vector with matrices as its elements. Note that they are notated with calligraphic capital letters in this article.
A 4D tensor can be thought of as a four-dimensional list of 3D tensors:
Another way to think about a 4D tensor is a vector with 3D tensors as its elements.
These can become more and more complex, but this is the extent which is necessary to proceed to performing operations with tensors.
Vector Operations
Assume these are vectors of the same length, i.
The operations that follow are mostly element-wise. This means that corresponding elements in each vector are operated on together.
Addition
import torch
x = torch.tensor([1, 3, 5])
y = torch.tensor([3, 7, 4])
x + ytensor([ 4, 10, 9])Subtraction
x = torch.tensor([1, 3, 5])
y = torch.tensor([3, 7, 4])
x - ytensor([-2, -4, 1])Dot Product (Multiplication)
This could also be notated with summation:
x = torch.tensor([1, 3, 5])
y = torch.tensor([3, 7, 4])
torch.dot(x, y) # 1*3 + 3*7 + 5*4 = 3 + 21 + 20 = 44tensor(44)This could also be performed with x @ y. The output of the dot product is a scalar. It does not return a vector.
Hadamard Product (Multiplication)
The Hadamard product is used to perform element-wise multiplication and returns a vector.
x = torch.tensor([1, 3, 5])
y = torch.tensor([3, 7, 4])
x * ytensor([ 3, 21, 20])Scalar Multiplication
k is a scalar, which is a real number in most instances.
k = 5
x = torch.tensor([1, 3, 5])
k * xtensor([ 5, 15, 25])Since k can be a fraction, this could also be considered scalar division.
Matrix Multiplication
Remember, a matrix is a collection of vectors. The same operations apply across vectors, but there are a few more rules to be cautious of when it comes to the rows and columns.
Assume the following, where X and Y have a shape of (4,3):
Addition
X = torch.tensor([[1, 3, 5],
[3, 6, 1],
[6, 8, 5],
[7, 3, 2]])
Y = torch.tensor([[6, 3, 2],
[8, 5, 4],
[3, 1, 7],
[1, 8, 3]])
X + Ytensor([[ 7, 6, 7],
[11, 11, 5],
[ 9, 9, 12],
[ 8, 11, 5]])Subtraction
X = torch.tensor([[1, 3, 5],
[3, 6, 1],
[6, 8, 5],
[7, 3, 2]])
Y = torch.tensor([[6, 3, 2],
[8, 5, 4],
[3, 1, 7],
[1, 8, 3]])
X - Ytensor([[-5, 0, 3],
[-5, 1, -3],
[ 3, 7, -2],
[ 6, -5, -1]])Dot Product (Matrix Multiplication)
When performing matrix multiplication, it is important that matrices are viewed as being composed of vectors. With this view, it is clear how the dot product is performed across the matrices. The only way multiplication can occur is if the number of rows in the first matrix match the number of columns of the second. This results in:
- (m, n) x (n, r) = (m, r)
If this is not the case, one of the matrices must be transposed to fit this order; this switches the rows and columns but preserves the vectors for the dot product.
In the image above, it is clear that each vector, or row, in the left matrix is multiplied by each vector, or column, in the second matrix. Thus, in this example, each vector in A has to be multiplied by each vector in B, resulting in 16 dot products being performed.
X = torch.tensor([[1, 3, 5],
[3, 6, 1],
[6, 8, 5],
[7, 3, 2]])
Y = torch.tensor([[6, 3, 2],
[8, 5, 4],
[3, 1, 7],
[1, 8, 3]])
X.matmul(Y.T) # X @ Y.Ttensor([[ 25, 43, 41, 40],
[ 38, 58, 22, 54],
[ 70, 108, 61, 85],
[ 55, 79, 38, 37]])Hadamard Product (Matrix Multiplication)
The Hadamard product is element-wise multiplication, so it is performed in the same manner as addition and subtraction.
X = torch.tensor([[1, 3, 5],
[3, 6, 1],
[6, 8, 5],
[7, 3, 2]])
Y = torch.tensor([[6, 3, 2],
[8, 5, 4],
[3, 1, 7],
[1, 8, 3]])
X * Ytensor([[ 6, 9, 10],
[24, 30, 4],
[18, 8, 35],
[ 7, 24, 6]])Scalar Multiplication
k = 5
X = torch.tensor([[1, 3, 5],
[3, 6, 1],
[6, 8, 5],
[7, 3, 2]])
k * Xtensor([[ 5, 15, 25],
[15, 30, 5],
[30, 40, 25],
[35, 15, 10]])Tensor Operations in Three Dimensions
Tensor operations require both tensors to have the same size unless the dot product is being performed.
For element-wise operations in this section, assume both tensors have a shape of (3, 3, 2). This means both tensors contain three (3,2) matrices. This can be seen below:
import torch
gen = torch.Generator().manual_seed(2147483647)
X = torch.randint(0, 10, (3, 3, 2), generator=gen)
Y = torch.randint(0, 10, (3, 3, 2), generator=gen)X = tensor([[[1, 4],
[9, 2],
[3, 0]],
[[4, 6],
[7, 7],
[1, 5]],
[[6, 8],
[1, 4],
[4, 9]]])
Y = tensor([[[8, 0],
[3, 6],
[6, 0]],
[[9, 1],
[0, 0],
[2, 2]],
[[7, 1],
[8, 1],
[9, 0]]])Addition
Addition is element-wise and performs as expected. The corresponding elements in each tensor are added to each other.
X + Ytensor([[[ 9, 4],
[12, 8],
[ 9, 0]],
[[13, 7],
[ 7, 7],
[ 3, 7]],
[[13, 9],
[ 9, 5],
[13, 9]]])Subtraction
Subtraction is also element-wise and performs as expected.
X - Ytensor([[[-7, 4],
[ 6, -4],
[-3, 0]],
[[-5, 5],
[ 7, 7],
[-1, 3]],
[[-1, 7],
[-7, 3],
[-5, 9]]])Dot Product (Tensor Multiplication)
Tensor multiplication is a little more complicated than in two dimensions. Before, the matrix multiplication could only occur if the follow condition was met:
- (m, n) x (n, r) = (m, r)
In three dimensions, this is still a requirement. However, the first axes must be the same:
- (z, m, n) x (z, n, r) = (z, m, r)
Why is this? Well, as previously mentioned, the dot product in two dimensions was focused on multiplying the vectors by each other. In three dimensions, the focus is on multiplying matrix-wise and then performing the dot product on each of the vectors within these matrices.
The image above should help explain this. It may help to think of both 3D tensors as a vectors of matrices. Since the dot product is performed by multiplying element-wise and then summing, the first thing that happens is each matrix is multiplied by its corresponding matrix. When this occurs, matrix multiplication results in every vector within the matrices performing dot products with the others. It is like a nested dot product in a sense.
For 𝓧 and 𝓨 to be multiplied by each other, the second and third axes of 𝓨 have to be transposed. 𝓧 and 𝓨 both have a size of (3, 3, 2). This means 𝓨 must become (3, 2, 3). This can be accomplished using Y.permute(0, 2, 1), which transposes the second and third axes. Alternatively, Y.transpose(1,2) could be used.
print(Y.transpose(1,2))
print(Y.transpose(1,2).shape) # Y.permute(0, 2, 1)tensor([[[8, 3, 6],
[0, 6, 0]],
[[9, 0, 2],
[1, 0, 2]],
[[7, 8, 9],
[1, 1, 0]]])
torch.Size([3, 2, 3])With the appropriate sizes, tensor multiplication can now be performed using either matmul or @. The output should have a shape of (3, 3, 2) x (3, 2, 3) = (3, 3, 3).
X.matmul(Y.transpose(1,2)) # X @ Y.transpose(1,2)tensor([[[ 8, 27, 6],
[72, 39, 54],
[24, 9, 18]],
[[42, 0, 20],
[70, 0, 28],
[14, 0, 12]],
[[50, 56, 54],
[11, 12, 9],
[37, 41, 36]]])Hadamard Product (Tensor Multiplication)
The Hadamard product performs as expected and multiplies element-wise across the matrices in the 3D tensor.
X * Ytensor([[[ 8, 0],
[27, 12],
[18, 0]],
[[36, 6],
[ 0, 0],
[ 2, 10]],
[[42, 8],
[ 8, 4],
[36, 0]]])Scalar Multiplication
Scalar multiplication also performs as expected.
k = 5
k*Xtensor([[[ 5, 20],
[45, 10],
[15, 0]],
[[20, 30],
[35, 35],
[ 5, 25]],
[[30, 40],
[ 5, 20],
[20, 45]]])Tensor Operations in Four Dimensions
Tensor operations in four dimensions still require both tensors to have the same size.
For this section, assume 𝓧 and 𝓨 have a shape of (2, 3, 3, 2). This means both 4D tensors contain two 3D tensors, and each of these contain three (3,2) matrices. This can be seen below:
import torch
gen = torch.Generator().manual_seed(2147483647)
X = torch.randint(0, 10, (2, 3, 3, 2), generator=gen)
Y = torch.randint(0, 10, (2, 3, 3, 2), generator=gen)X# 4d tensor
tensor([
# 3d tensor
[
# matrix 1
[[1, 4],
[9, 2],
[3, 0]],
# matrix 2
[[4, 6],
[7, 7],
[1, 5]],
# matrix 3
[[6, 8],
[1, 4],
[4, 9]]],
# 3d tensor
[
# matrix 1
[[8, 0],
[3, 6],
[6, 0]],
# matrix 2
[[9, 1],
[0, 0],
[2, 2]],
# matrix 3
[[7, 1],
[8, 1],
[9, 0]]]])Y# 4d tensor
tensor([
# 3d tensor
[
# matrix 1
[[6, 3],
[6, 4],
[5, 7]],
# matrix 2
[[3, 7],
[2, 0],
[5, 1]],
# matrix 3
[[0, 4],
[0, 8],
[0, 6]]],
# 3d tensor
[
# matrix 1
[[0, 5],
[8, 6],
[1, 0]],
# matrix 2
[[5, 1],
[1, 6],
[1, 2]],
# matrix 3
[[6, 0],
[2, 6],
[1, 5]]]])Addition
Addition, subtraction, and the Hadamard product are still performed element-wise as expected.
X + Ytensor([[[[ 7, 7],
[15, 6],
[ 8, 7]],
[[ 7, 13],
[ 9, 7],
[ 6, 6]],
[[ 6, 12],
[ 1, 12],
[ 4, 15]]],
[[[ 8, 5],
[11, 12],
[ 7, 0]],
[[14, 2],
[ 1, 6],
[ 3, 4]],
[[13, 1],
[10, 7],
[10, 5]]]])Subtraction
X - Ytensor([[[[-5, 1],
[ 3, -2],
[-2, -7]],
[[ 1, -1],
[ 5, 7],
[-4, 4]],
[[ 6, 4],
[ 1, -4],
[ 4, 3]]],
[[[ 8, -5],
[-5, 0],
[ 5, 0]],
[[ 4, 0],
[-1, -6],
[ 1, 0]],
[[ 1, 1],
[ 6, -5],
[ 8, -5]]]])Dot Product (Tensor Multiplication)
In four dimensions, tensor multiplication will have the same requirements as in three and two dimensions. It will also require the first and second axes to match for both tensors:
- (c, z, m, n) x (c, z, n, r) = (c,z,m,r)
In three dimensions, matrix-wise multiplication was performed, and it was followed by the dot product between vectors. The same steps will occur in four dimensions, but it will start with multiplying each 3D tensor by its corresponding 3D tensor. Then, each of their matrices will be multiplied by each other. Finally, their vectors will perform dot products on each other. This can be seen in the image above.
For this example, 𝓧 and 𝓨 have a size of (2, 3, 3, 2). For multiplication to occur, the third and fourth axes of 𝓨 must be transposed. This can be done the same way as before using Y.permute(0, 1, 3, 2) or Y.transpose(2,3). This transposes 𝓨 to have a shape of (2, 3, 2, 3).
The result should have a shape of (2, 3, 3, 2) x (2, 3, 2, 3) = (2,3,3,3). This means there will be two 3D tensors, and each of them will contain three (3,3) matrices. This outcome can be acquired using matmul or @.
X.matmul(Y.transpose(2,3)) # X @ Y.transpose(2,3)tensor([[[[18, 22, 33],
[60, 62, 59],
[18, 18, 15]],
[[54, 8, 26],
[70, 14, 42],
[38, 2, 10]],
[[32, 64, 48],
[16, 32, 24],
[36, 72, 54]]],
[[[ 0, 64, 8],
[30, 60, 3],
[ 0, 48, 6]],
[[46, 15, 11],
[ 0, 0, 0],
[12, 14, 6]],
[[42, 20, 12],
[48, 22, 13],
[54, 18, 9]]]])Hadamard Product (Tensor Multiplication)
X * Ytensor([[[[ 6, 12],
[54, 8],
[15, 0]],
[[12, 42],
[14, 0],
[ 5, 5]],
[[ 0, 32],
[ 0, 32],
[ 0, 54]]],
[[[ 0, 0],
[24, 36],
[ 6, 0]],
[[45, 1],
[ 0, 0],
[ 2, 4]],
[[42, 0],
[16, 6],
[ 9, 0]]]])Scalar Multiplication
k = 5
k * Xtensor([[[[ 5, 20],
[45, 10],
[15, 0]],
[[20, 30],
[35, 35],
[ 5, 25]],
[[30, 40],
[ 5, 20],
[20, 45]]],
[[[40, 0],
[15, 30],
[30, 0]],
[[45, 5],
[ 0, 0],
[10, 10]],
[[35, 5],
[40, 5],
[45, 0]]]])Please don’t forget to like and follow for more! :)
