Interesting ways to work with tensors in PyTorch
Determining tensors’ characteristics
PyTorch is an interesting library that permits to conduct many operations on tensors and permits also to use them to make optimizations for Machine Learning and Deep Learning. In order to give insight about useful manipulations that PyTorch permits to do on tensors, we selected 5 functions from PyTorch official documentation to present them and explain how they work. We also provide cases where these functions do not work. The selected functions are as follows:
- torch.diag()
- torch.matrix_power()
- torch.inverse()
- torch.eig()
- torch.matrix_rank()
To work with PyTorch library, the first thing we have to do is to import torch (in Python through Jupyter notebook in case we are using Python).
Section 1- The function torch.diag()
The first function we are talking about here is torch.diag(). This function permits to determine the diagonal of a 1-D or 2-D square tensor. In case we have a 1-D tensor (a vector), the function will provide a 2-D square tensor with the elements of the 1-D tensor as diagonal. The function will not work for a 3-D or more tensors. Below we have an example implementing the function:
We first created a tensor named a and used the function torch.diag() to get the diagonal of a. The output is the tensor b which contains the elements of the diagonal of the tensor a.
To provide more illustration, we have a second example in which the code is also working (without coding error message):
As we can see in this second example, the function torch.diag() extracts the elements on the diagonal of the tensor c. This function is very helpful if we want to extract the diagonal for a tensor. Nevertheless, we have to make sure that the tensor is 1-D or 2-D.
The tensor e is 3-D explaining why the function tensor.diag() gave an error message. As we have mentioned at the beginning of this section 1, a tensor has to be a 1-D or 2-D so that the function can work correctly.
To close this Section 1, we can say that the function torch.diag() can be used as long as we have a 1-D or 2-D tensor.
Section 2- The function torch.matrix_power()
Sometimes, we may want to get a tensor raised to a specific power n. For example, we may need to get the tensor raised to a power of n = 2 or more. In this case, the function torch.matrix_power() can be used. Note that if the value of n is negative, the function will return the inverse of the tensor raised to the power n. Also, the matrix has to be a square matrix with size (m,m).
We have created a tensor named a2 with size (4,4) and we computed the square of that tensor, b2, by using the function torch.matrix_power().
Another illustration:
As in Example 1 of this Section 2, we used the function torch.matrix_power() to illustrate its role by considering the tensor c2. Considering c2 and the value 3 as arguments for the function, we obtained another tensor d2 which is c2 raised to the power 3.
Now, to illustrate the case where the function provides error message, we give the below example.
We created a non-square tensor e2 and considered it as an argument in the function. As we can see, the function is not working here. We have to make sure that the tensor we are using as an argument is a square tensor. e2 is not a square tensor explaining why the code breaks.
The function torch.matrix_power() can be used only for a square tensor. Also, we have to make sure that the values in the tensor have a float data type.
Section 3- The function torch.inverse()
If we want to compute the inverse of a particular tensor, we can do that by using the function torch.inverse(). The inverse of a tensor can be useful in many tensors’ operations. The first illustration for this function is presented below.
In this example, we considered the tensor a3 and we used it as input in the function torch.inverse() to obtain its inverse. The output is b3 and it provides the inverse of the tensor a3.
In the Example 2, we did another illustration of tensor’s inverse calculation by using the function torch.inverse(). We considered c3 as an argument and we generated d4 which is the inverse of c3.
As we can see in the Example 3, the function provided an error message. This is because e3 is a singular linear matrix meaning that the rows or the columns are perfectly correlated. We say that e3 is not an invertible matrix.
To be able to use the function torch.inverse(), we have to make sure that the tensor we are using is a square tensor but also the tensor is invertible. Note that the function can be used for batches of 2-D square tensors. In that case, it will return a tensor composed with individual inverse for each batch.
Section 4- The function torch.eig()
We can also use PyTorch to determine the eigenvalues and eigenvectors for tensors by considering the function torch.eig().
In this Example 1 of Section 4, we computed the eigenvalues and the eigenvectors for the tensor a4. When we have the argument “eigenvectors=False”, the function will return an empty tensor matrix of eigenvectors as illustrated in the next example.
In this Example 2, we considered the argument “eigenvectors=False”. Using the function torch.eig(), we obtained the eigenvalues for the tensor c4 with an empty matrix of eigenvectors.
Example 3 of this section provides the case where the function returns an error message. We tried to compute the eigenvalues of the tensor e4 but the function did not accept the argument. The function torch.eig() to work has to get as input a square tensor. The tensor e4 not being a square tensor, its eigenvalues and eigenvectors cannot be calculated.
The function torch.eig() can be used only if we have a square tensor and if the tensor is not a singular linear matrix.
Section 5- The function torch.matrix_rank()
Sometimes, we may be interested in calculating the rank of a tensor. The rank of a tensor indicates the number of vectors in the tensor that are linearly independent. The function torch.matrix_rank() permits to determine a rank for a particular 2-D tensor by using SVD (Singular Value Decomposition) by default . We have below the first example for this case.
We used the function torch.matrix_rank() to determine the rank of the tensor a5. The output is tensor(4), meaning that the four vectors forming a5 are all independent.
Example 2 of Section 5 gives us another illustration to determine the rank. We calculated the rank of the tensor c5 which has size (2,10). The output is tensor(2) meaning that two columns of the tensor c5 can be written as a linear function of the other columns.
In the last Example of this Section 5, we considered a 3-D tensor e5 as input in the function torch.matrix_rank(). As expected, we got an error message.
We conclude that the function torch.matrix_rank() works only for a 2-D tensor. Also, note that this function works only if the values in the tensor are floating types.
Conclusion
In this document, we studied five functions from PyTorch and illustrated how to use them by providing some examples for each case. We provided also cases where the functions display error messages.
The functions we illustrated are torch.diag(), torch.matrix_power(), torch.inverse(), torch.eig() and torch.matrix_rank().
All these functions permit to provide useful information about tensors. Nevertheless, we have to be careful in providing the correct argument according to the considered function.
Reference links
- Official documentation for
torch.Tensor: https://pytorch.org/docs/stable/tensors.html - https://mathworld.wolfram.com/Tensor.html
