The link between Maximum Likelihood Estimation(MLE)and Cross-Entropy

Source: [5]

Maximum Likelihood Estimation

Maximum Likelihood Estimation(MLE) is a method to solve the problem of density estimation to determine the probability distribution and parameters for a sample of observations[2]. In MLE, the estimate computed by maximising the likelihood function[2].

Then Maximum likelihood estimation considers this problem as an optimisation problem/search problem where it looks for parameters represented as θ that best fit for the joint probability of observed data or best explain the observed data[2].

Likelihood function that measures the goodness of fit of a statistical model to a sample of data for given values of the parameters θ. Likelihood function, as shown below:

source:[4]

Where M is the number of classes in the classification problem.

Above equation indicates that Likelihood function is calculated by multiplying the discrete probability distribution over the classes raise to the probability of ground truth that is one hot encoded(It has all the probability mass on a single class or label).

For example:

source:[4]

For mathematical convenience, instead of product, we can add logarithm that is an increasing function[3]. This addition converts the equation to log-likelihood, as shown below[4]:

source:[4]

source:[4]

Mathematically, it is easier to minimise the negative log-likelihood function than maximising the direct likelihood[1]. So the equation is modified as:

Cross-Entropy

For a multiclass classification problem, we use cross-entropy as a loss function. An intuitive explanation of cross-entropy is the average bits of information required to identify an event drawn from the estimated probability distribution f(x), rather than the true distribution g(x)[6]. The lower the bit higher the approximation of f(x). When we use the cross-entropy loss function, the idea of reducing the bit using the optimisation process.

source:[4]

If we plug:

Then, we can identify that cross-entropy is equal to negative log-likelihood.

Source: [4]

If you like my write up, follow me on Github, Linkedin, and/or Medium profile.

References

1. https://amoudgl.github.io/blog/policy-gradient/
2. https://machinelearningmastery.com/what-is-maximum-likelihood-estimation-in-machine-learning/
3. http://statweb.stanford.edu/~susan/courses/s200/lectures/lect11.pdf
4. https://glassboxmedicine.com/2019/12/07/connections-log-likelihood-cross-entropy-kl-divergence-logistic-regression-and-neural-networks/
5. https://gombru.github.io/2018/05/23/cross_entropy_loss/
6. https://machinelearningmastery.com/cross-entropy-for-machine-learning/