Machine Learning Basics-The Norms

Norm is defined as a function by which we can measure the magnitude of a vector .Using norm we can measure how “large” a vector is. Norm maps a vector to non-negative value.The norm of a vector x is actually the measurement of the distance of the origin to the point x

More rigorously, a norm is any func with the following properties:

Properties of Norms

1. Norms are non-negative values.Norms are actually length or distance.So it can not be a negative number.But if a vector is zero, then norm can be zero.
2. Norms respect the triangle inequality.It just means that the norm of the sum of some vectors is less than or equal to the sum of the norms of these vectors: ||x+y||≤||x||+||y||
3. ||a⋅x||=||a||⋅||x||. Here, a is a scalar and x a vector. The norm of a vector multiplied by a scalar is equal to the absolute value of this scalar multiplied by the norm of the vector.

Application of Norms

1. Whenever we need to calculate the loss function, we need to know the magnitude of the distance between the predicated and actual values.
2. In regularization, we need to know the magnitude of weights.
3. In support-vector machine, we have to find the hyperplane that has the largest perpendicular distance between the hyperplane and the closest samples on either side. Here, we use norm to calculate these distances.

With respect to machine learning Norms can be classified as follows-

1. Lᵖ Norm
2. L¹ Norm(Manhattan/Taxicab Distance)
3. The Euclidean Norm(L² norm)
4. The Max Norm
5. The Frobenius Norm

Lᵖ Norm

Formal Definition of Lᵖ Norm

For p∈ℝ, p≥1, the p-norm is a norm on suitable real vector spaces given by the pth root of the sum (or integral) of the pth-powers of the absolute values of the vector components.

Using the following code Lᵖ Norm can be implemented-

# Import Numpy package and the norm function
import numpy as np
from numpy.linalg import norm


# Define a vector
v = np.array([2,3,1,0])

# Take the q-norm which p=2
p = 2
v_norm = norm(v, ord=p)

# Print values
print('The vector: ', v)
print('The vector norm: ', v_norm)

The output will look like this-

The vector:  [2 3 1 0]
The vector norm:  3.7416573867739413

L¹ Norm

Formal Definition of L¹ Norm

If p=1, we simply have the sum of the absolute values. It is what we have used intuitively at the beginning of this tutorial:

The Euclidean Norm(L² norm)

The L² norm is calculated as the square root of the sum of the squared vector values.theL¹ norm is more robust than the L² norm. This means that the L² norm is more sensible to outliers since significant error values will give enormous squared error values.

The max norm

Formal Definition of L¹ Norm

L∞ norm corresponds to the absolute value of the greatest element the vector.

This is the python code of Max Normal-

# max norm of a vector
from numpy import inf
from numpy import array
from numpy.linalg import norm
v = array([1, 2, 3])
print(v)
maxnorm = norm(v, inf)
print(maxnorm)

Running this code you should get the following output-

[1, 2, 3]
3.00

The Frobenius Norm

Formal Definition of Frobenius Norm

This is equivalent to take the L² norm of the matrix after flattening. L² norm is for vectors while Frobenius is for matrices.

max norm of a vector
from numpy import inf
from numpy import array
from numpy.linalg import norm
v = array([1, 2, 3])
print(v)
np.linalg.norm(v)

you should get the following output-

8.3666002653407556

Thanks!

That’s all for today. Don’t forget to this story with your friends and give us your 👏 !