📏 Quantifying Misalignment — A Data Scientist’s Intro To Information Theory — Part 3/5: Relative-Entropy

Gain intuition into Cross-Entropy, KL-Divergence and master their applications in Machine Learning and Data Analysis. Python code provided. 🐍

Photo by Immo Wegmann on Unsplash

[Misalignment enables us to assess] the amount we can win from a casino game, if we know the true game distribution is p but the house incorrectly believes it to be q. — Callum McDougall¹

This is the third article in a series on information quantification — an essential framework for data scientists. Learning to measure information unlocks powerful tools for improving statistical analyses and refining decision criteria in machine learning.

In this article we focus on estimating deviations of a prediction distribution from a ground truth as expressed in two commonly used metrics that are highly related: cross-entropy — a measure of misalignment — and Kullback–Leibler divergence — a distance measure (KL divergence, also known as relative-entropy).

Both of these are derivations of entropy, the focus of the second article, which quantifies…