Part 14 : Dot and Hadamard Product

Other than the matrix multiplication discussed earlier, vectors could be multiplied by two more methods : Dot product and Hadamard Product. Results obtained from both methods are different.

Dot Product

The elements corresponding to same row and column are multiplied together and the products are added such that, the result is a scalar.

Dot product of vectors a, b and c

Unlike matrix multiplication the result of dot product is not another vector or matrix, it is a scalar.

Dot product of vector a and b

Order of vectors does not matter for dot product, just the number of elements in both vectors should be equal.

The geometric formula of dot product is

Here |a| and |b| are magnitude of vector a and b and they are multiplied with cosine of angle between vectors

Dot product is also called inner product or scalar product.

Projection of Vector

Assuming that we have two vectors c and d, subtended by angle, phi(Ф).

Now, the projection of vector c on vector d could be represented as

Vector c with subscript-ed d represents projection of vector c on vector d

We can conclude from figure that the projection is equal to the horizontal component of vector c with respect to the angle phi(Ф).

This is called scalar projection.To find the vector projection of vector c on vector d we have to multiply scalar projection with unit vector d.

Substituting value of unit vector d.

Thus, dot product could also be represented as

Projections have wide use in linear algebra and machine learning (Support Vector Machine(SVM) is a machine learning algorithm, used for classification of data).

Hadamard Product (Element -wise Multiplication)

Hadamard product of two vectors is very similar to matrix addition, elements corresponding to same row and columns of given vectors/matrices are multiplied together to form a new vector/matrix.

It is named after French Mathematician, Jacques Hadamard.

Hadamard product of vector g, h and m

The order of matrices/vectors to be multiplied should be same and the resulting matrix will also be of same order.

Hadamard product of Matrix G and Matrix H (both of order 2x3), gives another Matrix N

Matrix N is of same order as input matrices (2x3)

Hadamard product is used in image compression techniques such as JPEG. It is also known as Schur product after German Mathematician, Issai Schur.

Hadamard Product is used in LSTM (Long Short-Term Memory) cells of Recurrent Neural Networks (RNNs).

Additional Resources :

Dot Product

Vector Projection

Hadamard Product

Jacques Hadamard

Issai Schur

Use of Hadamard product in JPEG

LSTM

Read Part 15 : Orthogonality and four fundamental subspaces

You can view the complete series here
I’m now publishing at avni.sh

Connect with me on LinkedIn.