Penrose’s Graphical Notation
In the last tutorial, we had a quick refresh about Tensors, matrices and vectors in general. In this tutorial, we will try to represent Tensors using Penrose’s Graphical Notation.
Let M represent the following Matrix
We see that the element at index (2, 3) is 2.1. In this case, what we did was we moved down by 2 along the Y-direction, and by 3 along the X-direction. We can also see that the element at index (0,2) is 4. In this case, we travelled by 0 in the Y direction and by 2 in the X-direction. We see that any element in the Matrix can be given using 2 numbers, one referring to how many rows to travel down, and the other referring to how many columns to travel across.
Thus, we can refer to all matrices as,
The circle or the node represents the value of the matrix at the given index (i,j). Its “hands” (more commonly called axis) represent the indexing within a certain direction and are currently given the arbitrary value of “i” and “j”. In our case, the “left hand”, given the value “i”, represents how much we travelled down across the Y-direction. Likewise, the right hand, currently assigned “j”, represents how far we travelled across in the X-direction.
Thus, this essentially means that the below diagram represents the value 99 as it is indexing item (1, 2) inside of our Matrix M.
Note: The maximum we can travel down in the Y-direction is 4, and the maximum we can travel in the X-direction is 3. This means that the highest value that can be assigned to the left hand is 4 and for the right hand it's 3. The number of values that an axis can take is commonly called its degree. So the degree for the left hand in our case is 5, and the degree for the right hand is therefore 4 since the right hand can take all the values from 0 to 3.
This may seem simple, but it nicely generalises very easily to Tensors of a different number of dimensions. Lets first see how it looks like for a Vector.
Let A be a vector set to:
Then the corresponding representation of A would be:
Where “i” ranges from 0 to 3, so the degree of the axes is 4.
The reason we only see one “hand” in this case, is that in order to specify a certain element in a vector, we only need to specify how far along one particular direction.
Example, in our case, for vector A, we see that the element at index 0 is -567. The element at index 3 is 1.2. In Penrose’s representation, this essentially means:
As we can see, specifying particular elements, within our vector just means assigning our “i” component with the index value.
Thus in general, any Vector A can be represented in Penrose graphical notation by:
Any Matrix A can be represented by:
Where “i” loops over the rows of A, and “j” loops over the columns of A.
These ideas will become useful when working with Tensors and Tensor contractions, which we will see later on. These diagrams are used extensively in physics, chemistry and engineering.
Finally, since a scalar is just a single number, it requires no further indexing, which means in graphical notation it is just a single lone Node:
Thus we see that:
Scalar product
We will now see how powerful this representation of tensors can be.
Let’s say we have two vectors, A and B.
Where,
The appropriate representation for the vector A in Penrose’s graphical notation would then be:
So if i=2,
The appropriate representation of B would be:
So if j = 3, B[j] = B[3] = -2
The dot product which we discussed in the previous article then gives us
In Penrose’s graphical notation, a dot product can be neatly represented as:
The two nodes joining together means that we are multiplying the value they represent. The “two hands” joining to become “one single hand”, means that given component “i”, we index into both the vectors and multiply that value together.
So, for example:
When two edges join together, that edge disappears and forms a new node that is given the value of the sum of all the products after joining the two edges. Thus we are left with:
Since C is the result of the sum of all the products of the elements of A and B after joining them together on a common edge, C is given by:
Notice that we have no free edges left, meaning that we are left with a scalar, which is exactly what the dot product gives us!
One important point to remember is that we can only join 2 axes when they both have an equal degree. Since both have their single axes to have the same degree, we can contract (or join) that edge.
Matrix Multiplication
Let A and B be the following matrices:
Likewise, matrix multiplication can also be visualised in graphical notation by:
Again with matrix multiplication, just like how we did for the dot product, we need to make sure that the axis on which we join have the same degree. For matrices, this ends up translating as the number of columns in the first matrix must be the same as the number of rows in the second matrix. This is because “j” indexes through the columns of the first matrix, but when matrix multiplied it will index through the rows of the second matrix forcing us to make sure that both the axes have the same size.
After the matrix axes have joined we are finally left with:
Over here, we see that the joining of the two axes formed a new node with its own two axes, which represents a matrix multiplication. We see that the axis “j” has disappeared but the axis “i” and “k” have remained. This makes sense since multiplying two matrices forms a new matrix with the same number of rows as the first and the same number of columns as the second, and this information is very clear in our diagram.
One important point to note is that it doesn't matter whether the axis “i” refers to moving along the Y-direction or the axis “j” refers to moving along the X-direction, so long as we remember to join two edges if and only if they have the same degree.
The trace of a matrix can then be represented as:
This is because, as we know the trace is defined to be the sum of all the elements (0,0), (1,1), (2,2) …., (i,j), where i=j
Thus as we sum across the diagonal of the matrix, we need the “i” component and the j component to be equal. In the Penrose’s graphical notation, this is given since the two axes are joined.
Once we've joined the axes together, the axis must now disappear. This means we are finally left with:
Which is a scalar, since it has no free edge, which is exactly what the trace of a matrix returns!!!
Tensor Contraction
Using Penrose’s graphical notation, we can also represent Tensor contraction very easily.
For example consider the Tensors, A and B, of size (4x8x2x7) and (8x7x4)
Now consider the following Tensor Contraction:
In essence, this basically means to find the result of the tensor contraction detailed above, we loop over all the “i” axis, then over the “l” axis then over the “j” axis.
The above formula looks incredibly complicated. However, in tensor contraction, it can easily be represented in a diagram notation as:
We can quite evidently see that the result of this Tensor contraction which occurred between 3 indices resulted in a vector, as there is only one free edge remaining. In this contraction, the “m” axis is joined with the “j” axis, the “n” axis was joined with the “l” axis, and finally, the “o” axis was joined with the “i” axis.
Again, it is important to note that:
Only then will the tensor contraction be possible.
Furthermore, we can now see how the matrix multiplication is essentially just a Tensor contraction between 2 second-order Tensors along one specific axis.
Tensor contraction along with the “i” and “j” axis only would produce a 3rd order tensor, and tensor contraction along the l axes only would result in a 5th order Tensor. This reason for this is left as an exercise to the reader, use graphical notation to your advantage.
Finally, a Tensor network, in essence, is basically just a bunch of Tensors where some Tensors may be connected with an edge, e.g
The above image is from Math3ma. For more information about Tensor networks in general visit Math3ma for more information.
In the next tutorial, we will view some of the key algorithms used to compute tensor networks in an efficient manner.
