Torch — Playing with the dimensions and shape of the tensor
A tensor, in the simplest terms, is an N-dimensional container. The torch library has many functions to be used with tensors that can change its size and dimensions.
Let’s look at some of them in detail -
To start with, let us import the required libraries.
1. repeat
This function returns the tensor repeated along the specified dimensions, like tiling.
torch.Tensor.repeat(*sizes)
sizes — torch.Size or int, that specifies the number of times each dimension has to be repeated.
The shape of the output tensor is an element-wise multiplication between the original shape and the input to the repeat function.
The memory location of the resulting tensor varies from the original tensor. Any changes in the output tensor are not reflected in its every tile.
The tensor is repeated once in dim 0 and twice in dim 1.
Shape of y: [2*1, 3*2] = [2, 6]
Notice that the storage size has changed. Both the input and output have different memory locations. Also, changing value at y[0][2] does not change the value of 5 at y[0][5].
The tensor is repeated 2 times along all 3 dimensions.
The shape of y tensor is calculated by taking the shape of x as [1, 2, 3]. Shape of y: [1*2, 2*2, 3*2] = [2, 4, 6]
- As normal multiplication intends, anything multiplied by 0 is 0. Similarly, specifying 0 in the repeats function call implies that the x tensor has to be repeated 0 times, resulting in an empty tensor.
- A tensor cannot have negative dimensions and thus the input to the function cannot be negative.
The number of dimensions in the input to the function must be equal to or greater than that of the original tensor.
2. repeat_interleave
This function returns the tensor obtained by repeating each item separately along the specified dimension rather than as a whole tensor.
torch.Tensor.repeat_interleave(repeat, dim)
repeat — int or torch.Tensor, specifying the number of times to be repeated.
dim — the dimension for expansion
Here too, the memory location of the resulting tensor varies from the original tensor. Any changes in the new tensor are not reflected in its every tile.
- When no dim is specified, the input is flattened, items repeated and thus the output is also flattened.
- When dim = 0 or 1, the items are repeated accordingly in the respective dimensions.
The tensor passed as an argument specifies the number of times each element in that dimension is to be repeated.
The error arises as the argument tensor has only two values that represent the repetition while the input tensor x has three values along dim 1.
3. expand
This function returns the tensor expanded along the mentioned singleton dimensions.
torch.Tensor.expand(*sizes)
sizes — torch.Size or int that indicates the desired size of the output.
The non-singleton dimensions will be retained in value and can either be written as is or as -1 while passing to the function.
The memory location of the output tensor is the same as the input and so, any changes in the input or output tensor will be reflected throughout.
The input tensor has size [1, 4]. The argument specifies that the tensor is to be expanded along dim 0 (singleton dimension) while dim 1 retains value 4.
Changing the value at y[0][0] is reflected throughout as their storage is the same.
The size of the input is taken to be [1, 1, 2, 2] rather than [2, 2] as the arguments to the function represent a 4-dimension tensor. Here, the singleton dimensions are dim 0 and dim 1, which will be expanded to 2, and the others are retained by specifying -1.
The original shape of the tensor needs to be retained as the last arguments passed to the function because any change in dimensions is appended to the front of the shape tensor.
[2, 2] → [d1, d2,…, dn, 2, 2]
4. reshape
The function returns the tensor in the specified shape with the elements and their count intact. The product of the tensor.Size() items will be equal for the input and output tensors.
torch.Tensor.reshape(shape)
shape — tuple of int containing the new shape. -1 can be used in place of a dimension to be inferred from the rest.
The missing dimension is figured out from the size of the original tensor. Elements in the original tensor = 6 * 1 = 6. Thus, missing dimension = 6/2 = 3.
The error arises because the argument [4, 2] requires 8 items in the original tensor while it has only 6.
5. squeeze
This function returns the tensor with all the dimensions with value 1 removed. This function can be used to remove unnecessary dimensions with value 1.
torch.Tensor.squeeze(dim, out)
dim — optional, the dimension along which it has to be squeezed
out — optional, output tensor
This does not change the memory storage.
The dim 3, 4 have been removed reducing the tensor from 5-dimensional to 3-dimensional.
The tensor has been reduced from 5-dim to 4-dim by removing only the dim 2.
The expected dimension is out of range.
These are just five of the many functions available in torch. Some of the others include
- flatten — this reduces the tensor to a single dimension.
- expand_as — this is the same as the expand function, but takes a tensor as an argument for the reference shape.
- repeat_as — this is the same as the repeat function, but takes a tensor as an argument for the reference shape.
- unsqueeze — this adds a dimension of size 1.
- view — this returns a view of the tensor with a different shape, works with contiguous tensors.
and others.
Each of these functions is explained further in the torch.Tensor official documentation.
