Tensorflow — what are tensors and how are they used in Machine Learning?
What is TensorFlow? For someone who has just started learning to code, TensorFlow seems like a foreign and cryptic topic. However, machine learning is something I definitely want to explore. I decided to search for some background information on TensorFlow and ended up in a very deep hole.
Google’s Brain team developed TensorFlow and uses it for research and production. It is an open source software or math library that aids in developing and training machine learning models. TensorFlow works with tensor data. OK… What does that mean and what does it have to do with machine learning?
When searching for tensors on the web, it is found to have existed for over a hundred years. Albert Einstein’s theory of general relativity was published in 1916 and entirely based on tensors. It took a great amount of effort for Albert Einstein to learn tensor analysis and it was more broadly accepted because of his famous theory. Below is the Einstein tensor that can be found in the Einstein field equations for gravitation and more information about it can be found here.
In TensorFlow, tensors are immutable multi-dimensional arrays that contain dtypes — objects representing the type of elements in tensors. In machine learning, the data that is processed is usually an object with dimensions. An example would be a photo. The photo has pixels and those pixels have intensity, location, and color/depth. Tensors are used to represent the photo’s data with their different dimensions. Generally speaking, if you break a tensor down to its different components, it consists of scalars, vectors, and matrices.
Before diving deep into tensors’ different parts, let’s first discuss what shapes and ranks are. Shapes describe the dimension/axis’s length of a tensor. A tensor with a shape [3 x 3] would therefore have 2 axes, each with a length of 3. Ranks represent the number of dimensions of a tensor.
Now, scalars, or rank-0 tensors, have the smallest dimensions of [1x1] and represent magnitude. They do not have any axes because they represent only a point so they have a shape of 0. By adding a second dimension, such as direction, the shape increases and the scalar is now a vector(rank-1 tensors). They have dimensions of [1x1x1x1] or M x 1, like a line. Each vector is a list of scalars. Following that thought, a matrix(rank-2 tensors) is composed of multiple vectors:
[1x1 … 1x1] [1x1 … 1x1] or M x n. When there are multiple layers of matrices, we arrive at our tensor (M x n x K), a multi-dimensional representation of our data. There can be different shapes when our data has more than 2 dimensions. Reshaping is possible and it affects the ordering of each element but not the actual elements themselves.
With tensors, we can add more characteristics by adding more layers of tensors. In the example of a photo, we can make any photos 3D simply through adding another layer of tensors or dimension to our photo’s tensors. As models get more complex, their dimensions change.
How are tensors and processing speeds related?
TensorFlow works with tensors and makes it easier not only for CPUs but also GPUs to calculate data. There was a CPU and GPU comparison done on training the Convoluted Neural Network or CNN. The comparison reports that processing data primarily with a GPU can result in 85% less training time than primarily using CPU. Google announced in May 2016 that they developed a TPU (tensor processing unit) for TensorFlow and have been using it internally for a year. With TPUs, the training time is dramatically reduced. The TPU chip is an AI accelerator application specific integrated circuit and it does not replace CPU and GPU as it was specifically created for neural network calculations. It can reduce a month’s worth of training down to a few days. The chip is proprietary and can only be accessed through the Google Cloud service.
Graphs
TensorFlow’s tensors are usually the result of computational data or input data. Graphs are sets of computations that are executed successively. They contain nodes that each represent an operation. These operations or nodes are all connected by edges or tensors that populate the operations with data. TensorFlow offers an extensive library of operations and allows customized operations as well.
As previously mentioned, tensors allow the addition of characteristics. An existing model can be fed a collection of feature vectors that represent the characteristics of interest. For example, a list of feature vectors represents people’s heights in centimeters and weights in kilograms respectively:
[ 154 57]
[172 63]
[180 68]
This information flows through the node and creates a new tensor that is then used for a new operation.
TensorBoard is a tool that allows users to create graph visualizations to understand and debug them as graphs become more and more complicated.
More to explore
As I researched more about TensorFlow and the impressive projects that have been built with it, I realized how much work I have to do. In addition to understanding the concept of tensors and how data flows in a graph, there is a plethora of other concepts such as supervised vs unsupervised learning, convolutional neural networks (CNN), deep learning and more. I want to start working on simple tasks such as differentiating between cat and dog photos through CNN and eventually even working on creating an ASL translation mobile application to help those in the deaf or mute community (I would probably have to graduate from only signing letters).
