Member preview

From Scalar to Tensor: Fundamental Mathematics for Machine Learning with Intuitive Examples Part 1/3

For understanding the mathematics for machine learning algorithms, especially deep learning algorithms, it is essential to building up the mathematical concepts from foundational to more advanced. Unfortunately, Mathematical theories are too hard/abstract/dry to digest in many cases. Imagine you are eating a pizza, it is always easier and more fun to go with a coke.

The purpose of this article is to provide intuitive examples for fundamental mathematical theories to make the learning experience more enjoyable and memorable, which is to serve chicken wings with beer, fries with ketchup, and rib-eye with wine.

The 3-course fundamental mathematics for machine learning meal is organized as follows:

From Scalar to Tensor: Fundamental Mathematics for Machine Learning with Intuitive Examples Part 1/3

  • What are Scalar, Vector, Matrix, and Tensor?
  • Addition between Scalar, Vector, and Matrix
  • Multiplication between Scalar, Vector, and Matrix
  • Identity and inverse matrix
  • Diagonal matrix and symmetric matrix

From Norm to Orthogonality: Fundamental Mathematics for Machine Learning with Intuitive Examples Part 2/3

  • 1-Norm, 2-Norm, Max Norm of Vectors
  • Orthogonal and orthonormal vectors
  • Orthogonal matrix

From Eigendecomposition to Determinant: Fundamental Mathematics for Machine Learning with Intuitive Examples Part 3/3

  • Eigendecomposition of matrix: eigenvalue and eigenvector
  • The trace operator
  • Determinant square matrix

In this article, we will go through the part 1/3, From Scalar to Tensor with intuitive examples.

What are Scalar, Vector, Matrix and Tensor

Scalar is a single number. Usually, we write scalars in italic and lowercase, such as “x”. Scalar could be any type of number, for example, natural number, rational number, or irrational number as shown below:

  • Natural number: 3, 4, 5, …
  • Rational number: 4/3, 1.666666…
  • Irrational number:

The following code shows how to use Numpy’s N-dimensional array ndarray to create a scalar variable.

import numpy as np

x = np.array(1.666666)

Vector is an array of numbers. Typically, we denote vectors with bold and lowercase letter, and put elements of a vector as a column enclosed in square brackets:

The following code shows how to generate a vector with Numpy.

x = np.array([1,2,3,4,5,6,7,8,9])

Matrix is a 2-D array of numbers, which usually denoted with bold and uppercase variable names, such as:

The following code shows how to create a 4*4 matrix with Numpy.

x = np.array([[1,2,3,4],[5,6,7,8],[9,10,11,12],[13,14,15,16]])

Tensor is a k-D array of numbers. The concept of tensor is a bit tricky. You can consider tensor as a container of numbers, and the container could be in any dimension. For example, scalars, vectors, and matrices, are considered as the simplest tensors:

  • Scalar is a 0-dimensional tensor
  • Vector is a 1-dimensional tensor
  • Matrix is a 2-dimensional tensor

Thereby, we can deduce that a 3-D tensor is a cube, 4-D tensor is a vector of cubes, 5-D tensor is a matrix of cubes, 6-D tensor is a cube of cubes, etc.

Photo credit to knoldus

We denote tensor in bold and uppercase letter, and identify the element of tensor at coordinates (i, j, k) by writing:

The following code shows how to create a 3-D tensor with Numpy.

x = np.array([[[1, 2, 3],[4, 5, 6],[7, 8, 9]],
[[10, 20, 30],[40, 50, 60],[70, 80, 90]],
[[100, 200, 300],[400, 500, 600],[700, 800, 900]]])
3-D tensor

In industry, commonly, we store time series data in 3-D tensor, image data in 4-D tensor, video data in 5-D tensor, etc.

Addition between Scalar, Vector, and Matrix

Scalar+Matrix = add the scalar to each element in the matrix:

Vector + Matrix = add the vector to each row in the matrix:

Matrix + Matrix = add their corresponding elements as long as the matrices have the same shape:

Multiplication between Scalar, Vector, and Matrix

Scalar・Matrix = each element of the matrix multiply the scalar

Matrix・Vector = dot product of each row in the matrix and the vector

Identity and inverse matrix

In order to calculate the inverse of a matrix, we need to build the concept of identity matrix. Identity matrix is simple from structure point of view: all the entries along the main diagonal are 1, while all other entries are 0, such as:

Identity matrix

The inverse of matrix A is denoted and defined as following:

The necessary and sufficient condition for the existence of the inverse of a matrix:

In the following example, the second column of the matrix could be considered as the first column multiply by 2, so that the matrix is linearly dependent, which means it does not have an inverse.

Diagonal matrix and symmetric matrix

A matrix is diagonal if and only if,

A square diagonal matrix whose diagonal entries are given by vector v can be denoted as diag(v), for example,

A symmetric matrix is any square matrix that is equal to its own transpose:

For example, the following 3*3 matrix is symmetric:

Congratulations! You just completed one-third of Fundamental Mathematics for Machine Learning with Intuitive Examples. It is straightforward to understand when there are examples, right?

Next step: From Norm to Orthogonality: Fundamental Mathematics for Machine Learning with Intuitive Examples Part 2/3