What are tensors? How are they used in Machine Learning.

Hello There people, In my previous article I mentioned a few points about tensors, and that I would be writing a separate article about it. As a person working with tensorflow, I had so much doubt about tensors. What are tensors? Why haven’t I ever heard of it? how long have they been existing? It took me long enough but after a lot of searching and reading I was able to come to an understanding of what they are and how they are useful. So here we go…

What are tensors?

This question on the internet will get you so much that the only thing you can understand will be“tensors are mathematical objects” or “algebraic objects” to be more specific. There will be no shortage of complex definitions though. So let me tell you what I could find. Tensors are a more generalized representation of vectors interacting in higher dimensions. They have two parameters called dimensions and rank. Yes to get a complete understanding we are going to have to step into the mathematics behind it. So let’s consider an example to make it easier.

Let us consider two vectors, with unit vector(otherwise called component vectors) i ,j and k. Three unit vectors indicates 3 dimensions. So the first parameter mentioned earlier, that is dimension takes the value 3. Now we are considering two vectors. So we can produce a tensor of rank 2. Rank 2 tensors is the basic that’s the reason I chose the rank as 2.

Now let’s get the two vectors

v1 = a1i + b1j + c1k and v2 = a2i + b2j + c2k

Consider performing an operation on this vector be it addition, multiplication or anything you want to do. The Resultant can be expressed as a vector. The cross product of two vectors is a vector. Similarly the sum and difference of two vectors are a vector. So we can obtain a single resultant when we think in vectors. This is where tensors change the game. Can you even guess how this can be converted into a matrix representation. Think of the possible ways you can come up with and then compare it with the proper way if you want to have some fun. Now let’s arrange our stuff, the vectors.

Consider this your hint. This is the arrangement we are going to use or the arrangement tensors use. Keep thinking on one side in parallel, on the other side let’s go back to the parameters. The dimension is three and rank is two. So the tensor needs dimension power rank elements to represent. In our example it is 3 power 2 which is 9. So this might give a little understanding for the arrangement. But still I know most of you are not clear. So let’s go crack it

The real need for tensor arises when the variance in vectors are high. We saw that the resultant is a vector but now consider that a1 is changed in the vector v1 such as a1 is increased by 4. Now to calculate the new resultant, we will have to again perform the operation with the new set of vectors. Because change in one component affects all the component of other vector so the resultant is entirely different. This is why we have to repeat the operation again.

Here is the tensor approach, In our matrix have all of the elements separately. So you can just map the change in the matrix, so go back to the example of a1 changing. This can be represented as…

And voila, There comes all the changes that could have brought up by a1. And Now the same way the changes brought up by all the different components can be represented the same way, then the representation can be completed as …

So when you read or hear tensors are the generalization of vectors, this is what it actually means. This example may make it look easy, like some simple calculation you did in school, but this is just a very simple example with a rank 2 tensor. It gets much complicated with higher ranked tensors moving to tensors of tensors at certain points. So if you were starting to think it’s easy you might want to drop that thought.

By now you should also understand that a rank 1 tensor is simply a vector. Because it cannot be generalized more. I guess that pretty much explains the basic of tensors.

Why haven’t we heard the word “tensor” this long?

It’s simply because you don’t really have any need of using or even knowing them unless you went into very high and dense field of physics or mathematics. Even in those fields you find it when you doing a very high level of graduation. Just for mentions there are used in study of Einstein’s theory of relativity, fluid dynamics any other study which involves a very big or complex coordinate system. So you can bring in fields like dark matter research or string theory and very other complicated physics theories and hypothesis. So you really are interested in these domains then you may get to learn about tensors in college.

How are they used in programming ?

Tensors became well known to IT after google’s machine learning framework tensorflow. They use tensors as the basic unit for calculation. Although the term is same, they are not entirely same. Tensors in programming are not the same as tensors in mathematics. They just inherit some of their qualities. They take some representation techniques from them.

Here they are just represented as arrays of arrays or lists of lists. In machine learning there are lot of ways these representation can be manipulated and they do not have to obey the strict coordination transformation laws put forth by mathematics and physics. So it cannot be considered entirely equivalent to the mathematical tensors but can be considered to inherit some of those mathematical properties.

Tensorflow has an attribute called shape to represent every tensor. Every tensor has a shape(x,y) where x is the length of the tensor and y is the dimension of matrices or list/array in this case inside the tensor. It has to be equal for every list/array inside.

Why sudden fascination for tensors in machine learning and deep learning?

Tensors use matrix to represent. It makes it so much easy to represent information in an array. Consider a image of some resolution Y x Y. The pixel data can of the images can be so easily represented in an array. The same can be said for frames in a video. The representation becomes way easier. So the important takeaway we can get such accurate representation of data such that it can be considered almost heavily close to the natural representation of those objects.

This is why tensorflow enjoys popularity among developers and with many other companies adopting it when there are so many other frameworks available.

Conclusion

So that’s all I got about tensors. These are basic understanding that is good to have when you work with tensors. Hope this article helped you expand your vision and you can expect more on write ups on tensorflow, tutorials and other stuff also in other domains from me. And hey if you got the right idea for the arrangement and representation, give yourselves a clap, you deserve it.