[ Archived Post ] Multivariate Gaussian distributions and entropy 3

3 min readJan 20, 2019

Please note that this post is for my own educational purposes.

The classic normal distribution → the formula as well as what the standard deviation. We can even do MLE → by just taking the mean of the data as well as variance.

A higher order of normal distribution → what we can achieve when we use vectors

When the distribution is dependent we would have a skewed, else the normal distribution would only change in the x and y-direction. (The MLE also follows this case → just the mean of the vector.).

When the distribution is independent of one another → So we get the visual that it is only changing in the x and y-direction, not a skewed direction. (this is very useful and interesting). (joint distribution just the multiplication between two distribution).

We can understand this as eigenvalue decomposition as well. The red curve is given by the constant calculated above → and the sigma can be decomposed using eigenvalue decomposition.

How each of the components is scaled to one another. (did not know that we can understand this as a geometric point of view.).

How the decomposition works in Matlab.

The distribution is independent of one another → that is what we get.

But now there is dependec → and we can see the skew.

Now we are moving the distribution.

The differential entropy is not the continuous version of discrete entropy → in the process of conversion we lose some property such as the entropy being negative and more.