Intuitive Understanding of Tensors in Machine Learning

Tensor is an important concept in many scientific fields, such as mathematics, physics, signal processing, and computer vision, to name just a few. In mathematics, a tensor is an algebraic object that describes a multilinear relationship between sets of algebraic objectives related to a vector space in a meaningful way (see a Wikipedia description: https://en.wikipedia.org/wiki/Tensor). There are many types of tensors, including scalars, vectors, matrices, and higher-order tensors. Strictly speaking, scalars, vectors, and matrices are the simplest tensors, as we do not use the “tensor” concept to mention them.

Figure 1. Illustration of scalar, vector, matrix, and third-order tensor. If one use the concept of dth-order tensor, then d=1 refers to the vector, d=2 refers to the matrix, and d=3 refers to third-order tensor.

As shown in Figure 1, each small cubic indicates a certain entry. In general, vector can represent a list of entries (usually in a column), and matrix can represent a table of entries with rows and columns. By contrast, a higher-order tensor (e.g., third-order tensor) may be represented as a multidimensional array with multiple indices. In what follows, we introduce the components of third-order tensors — including entries, fibers, and slices — via the use of some intuitive illustrations.

Components of Third-Order Tensors

Although the tensor concept what we mention in this story is seemingly different from both vector and matrix, it is not hard to describe the tensor as a…