Vector Norms: Introduction
What is a norm of a vector? Why is it used in machine learning? Let’s see in this post.
Vector norm is a function that returns the length or magnitude of a vector. It has many applications in Machine learning, some of them are,
· Evaluate the error of the model (difference between predicted and actual value).
· Used to define the regularization term (to avoid the risk of overfitting).
Vector norms have following properties:
· Positivity — Vector norms are non-negative values (think of it as measure of distance!), vector norms are zero only if the vector itself is zero.
· Triangle inequality — The norm of the sum of two or more vectors is less than or equal to the sum of the norms of these vectors
· Linearity — for any scalar quantity k,
Few of the vector norm calculations used in machine learning are L1 norm, L2 norm etc.
Lp-Norm of Vector:
Norm is a kind of measure of size of mathematical object.
One interesting fact is ordering exists for rational numbers (real numbers), that is we can compare rational numbers easily,
For example, ‘7’ is larger than ‘4’, ‘0’ is larger than ‘-2’.
But what about complex numbers?
Can we say ‘3i’ is greater than ‘-2i’?
This is not correct, so from the knowledge of complex numbers, the modulus is used for comparing complex numbers.
If we have 2 complex numbers, x+iy and a+ib, most of the times we compare sqrt(x²+y²) and sqrt(a²+b²).
For a 3-D vector, |v| = sqrt(Vx²+Vy²+Vz²).
It leads to the idea that for understanding the size of a vector, we have Lp-norm of a vector.
Different vector norm calculations can be done by varying p value (1,2….).
Vector L1 Norm:
It is called Manhattan norm or taxicab norm; the norm is a calculation of the Manhattan distance from the origin of the vector space.
Manhattan distance is perhaps more useful to vectors that describe objects on a uniform grid, like a chessboard or city blocks. The taxicab name for the measure refers to the intuition for what the measure calculates: the shortest path that a taxicab would take between city blocks (coordinates on the grid). Mathematically, it is defined as below.
Vector L2 Norm:
It is called as Euclidian Norm after the famous Greek mathematician Euclid of Alexandria and it is the most commonly used norm. Defined as the square root of the sum of the squares of the components of the vector.
L-infinity Norm
It gives the absolute value of the largest element of the vector (Interested in proof? Click here), so it is also called Max norm.
Note:
What happens when p=0?
We may call it as L0 norm, but strictly speaking, it is not actually a Norm because it violates the homogeneous (Linearity) property of the norms.
It is a cardinality function which has its definition in the form of Lp-norm, though many people call it a norm.
It is used to calculate the number of non-zero elements in a vector, that is sparsity can be modelled using L0 norm.
Interested in calculating vector norm?
It can be calculated in Numpy using norm() function, for more details click here.
Happy Learning!!!
References:
