Linear Algebra in Artificial Intelligence & Quantum Computing
Linear Algebra Usage Introduction: Linear Algebra is the primary computation tool in both Artificial Intelligence (AI) and Quantum Computing (QC). In this blog will cover the required topics for storage and computing for both the fields.
Linear Algebra is basically a Mathematical Computational Tool and widely used in many areas like science , engineering and computational fields. Following diagram describes the concepts used in Linear Algebra for AI and Quantum Computing, first 4 topics (Vectors, Matrices, Tensors and Product Rules) covered in this article and the remaining topics are Decomposition , Advanced Concepts for Quantum Computing, Matrix Calculus for Machine Learning and Deep Learning in other articles.
Note: Decomposition and Matrix Calculus links updated soon.
Primarily your data has to be stored in some where else to compute, Linear Algebra stores your data in Vectors, Matrices and Tensors and then operate as per your instructions. Linear Algebra is a Storage Space.
Title diagram shows what are mathematical objects as well as AI sub fields and Quantum Computing uses Linear Algebra.
Artificial Intelligence: Linear Algebra can be applied in Machine Learning and Artificial Intelligence, including exposure to clustering, data fitting, classification, validation, and feature engineering.
Linear algebra applied in unsupervised learning methods (like k-means) such as principal component analysis (PCA), non-negative matrix factorization and more sophisticated clustering methods. You can apply regression and classification methods , such as logistic regression and the support vector machine, as well as methods for computing model parameters that scale to extremely large scale problems.
In the above Linear Algebra applied in Data fitting, feature selection and engineering, Machine Learning sub-field of AI shown pictorially.
Quantum Computing: Quantum Computing is the new computational model based on nature and celestial bodies, where as classical computers (Desktop, Laptop) works on Classical physics. QC requires hi-dimensional vector space and advanced Linear Algebra concepts like Tensor Product, Decomposition, Spectral Theorem, Matrix Exponential, etc., Below is the picture of Linear Algebra in Quantum Computing. Every computation requires Linear Algebra stuff. This link describe Quantum Computing — Required Linear Algebra (Advanced Concepts discussed).
Due to topics broad usage in AI and QC have discussed topics in separate articles.
In Linear Algebra 4 major Data Structures or Mathematical Objects and their properties , operations need to be understand. They are Scalar, Vector , Matrices and Tensors.
Scalar : It is simply a single number. It can be Real Number or Natural Number.
Vector: It is a list of numbers. The numbers are arranged in order. Identifying each element by its index. We can consider vectors as identifying points in space, with each element giving the coordinate along a different axis.
Matrices: A Matrix is a two dimensional array of numbers, each element is identified by two indices. For instance, the following is the Matrix.
Tensor: An array of numbers arranged in more than two axes. “In the general case, an array of numbers arranged on a regular grid with a variable number of axes is known as a tensor.” — DeepLearning Book
Tensors can simply taken as an N-dimensional Arrays, and it is Generic for Scalar, Vector, Matrices and Tensor.
For instance ; in 3-dimensional arrays it is denoted by
The first Axis denotes Rows, Second Axis denotes a list of rows stacked as columns, Third Axis denotes a list of columns stacked as levels in a cube.
Tensors are very important in Machine Learning, Deep Learning and Physics (Mechanics and Electrodynamics or General Relativity).
Transpose : It maps rows to columns and columns to rows and denoted by ‘T’. It is applicable in vectors and matrices.
Vector Norms: Measuring the size of a vector is called Norm. It is general form is given by :
If p=2 , then it is Euclidean norm. It is simply the Euclidean distance from the origin to the point identified by vector v. It can also be calculated as
If p=1 , will use in Machine Learning when the difference between zero and nonzero elements is very important.
Max Norm: It simplifies to the absolute value of the element with the largest magnitude in the vector.
Frobenius Norm: Measuring the size of the matrix use Frobenius Norm. It is defined as
Dot Product in terms of norms: The dot product of two vectors can be written in terms of norms. It is defined as
Complex Conjugate: The complex conjugate of each element is as follows:
Adjoint Vector: The adjoint of a vector v is denoted as
and is defined to be the following row vector where * denotes the complex Conjugate.
Product Rules in Linear Algebra
Products of Vector:
Inner Product or Dot Product: Given two vectors
called the inner product or dot product of the vectors, is a real number given by
Outer Product: Given two vectors u , v and their product defined is as follows
Scalar — Vector Product:
The remaining product rules comes in the respective sections.
Scalar Matrix Product:
Matrix: A Matrix with dimensions with ‘m’ rows and ’n’ columns and ’n’ columns can be defined is as follows.
Where as a Vector is defined as n-rows and 1-columns and vector is special case of matrix.
Hadamard Product: It is also known as element-wise or entry-wise or schur product. It is simple and binary operator works on 2 Matrices of same dimensions and produces output matrix whose dimensions are same as input matrices. If dimensions are not equal then Hadamard product is undefined. Below describing Notation & Hadamard Product.
Matrix Multiplication:Multiply two matrices A , B to get a new matrix ‘P’ of dimension of ‘mp’. For example the element of first row , first column in P as Inner product of first row of M with the first column of N. Like wise, second element in first row defined as Inner product of first row of m with the second M column of N.
In the same way we will define each element / entry of P as a inner product of row in first matrix & column in second matrix.
Tensor Product discussed in detail Quantum Computing and Linear Algebra article.
Matrix-Vector Products: There are two ways of looking at matrix-vector multiplication, one write A by rows, second write A by columns in two forms described in the below image.
Let’s write A in column form. In this case , y is a linear combination of the columns of A, where the coefficients of the linear combination are given by the entries of x.
Let’s write A in row form. The ‘i’th entry of y is equal to the inner product of the ‘i’th row of A and x.
Express A in terms of its Columns and resultant is as follows
Express A in terms of its Rows and resultant is as follows
Matrix Operations and Properties: There are various types of matrices exist in Linear Algebra for different usages.
Square Matrix: Whose Dimensions rows and columns are same.
Non-Square Matrix: Dimensions are different
Identity Matrix /Unit Matrix: Square Matrix, Whose Diagonal elements are ‘1’, Non-Diagonal elements are ‘0’.
It is also known as Unit Matrix and defined is as follows:
Diagonal Elements: Diagonal Element is the one whose indices are same.
Diagonal Matrix: A Diagonal Matrix is a Matrix whose non-diagonal elements are 0. It is denoted is as follows:
Inverse of a Matrix: For a Square matrix A, B is said to be inverse then
as well as defined as the matrix such that
Trace of a Matrix: For a Square matrix , sum of diagonal elements and defined is as follows.
Symmetric Matrix: A symmetric matrix is any matrix that is equal to its own Transpose. i.e
Anti-Symmetric: It is defined as negative with its own Transpose.
The Matrix A can be represented as a sum of Symmetric and an anti-symmetric matrix,
Singular Matrix: A matrix is Singular if and only if its determinant is 0.
Non-Singular Matrix: A Square matrix is Non-Singular if and only if its determinant is non-zero.
Linear Combination: A Linear Combination of some set of vectors
Linear Independence: A set of vectors
is said to be Linearly Independent if no vector can be represented as a linear combination of the remaining vectors.
Linear Dependence: If one vector belonging to the set can be represented as a linear combination of the remaining vectors, then the vectors are said to be Linear Dependent i.e.,
Orthogonal:Two Vectors are said to be Orthogonal to each if their inner product is 0.
Orthonormal: If the vectors are not only orthogonal but also have unit norm, the we will call Orthonormal.
Determinant: The Determinant of a Square Matrix , is a function mapping from matrices to real scalars. It can be denoted by det A or |A|.
Rank:
Column Rank: It is the largest subset of columns of A that constitute a linearly independent set. In other words, it is simply as the number of linear of linearly independent columns of A.
Row Rank:It is the largest number of rows of A that constitute a linearly independent set.
Rank: The Column rank of A is equal to the Row rank of A , and both quantities (column & row rank) are collectively referred to as the rank of A, denoted as rank(A).
Properties of Rank:
Span: The Span of a set of vectors is the set of all vectors that can be expressed as a linear combination.
In other words, any vector can be written as a linear combination of
Projection:
Range (or) Column Space: The range of a Matrix A denoted as R(A) and is the span of the columns of A.
Null Space:
Adjoint (or) Conjugate of a Matrix: For any Matrix ‘M’, Adjoint ’N’ is
Unitary Matrix: Unitary Matrices preserve norm of a Vector. Unitary Matrices are very important in Quantum Computing.
Hermitian Matrix: It is a complex square matrix that is equal to its own conjugate transpose.
The basic Linear Algebra stuff is covered in a separate article Linear Algebra in Artificial Intelligence and Quantum Computing article. It is better to refresh Linear Algebra stuff before you start, the required basic stuff of Linear Algebra covered there.
Thanks for reading. Drop a note for any mistakes, feedback and comments.
