Member-only story

🤷 Quantifying Uncertainty — A Data Scientist’s Intro To Information Theory — Part 2/4: Entropy

Gain intuition into Entropy and master its applications in Machine Learning and Data Analysis. Python code provided. 🐍

Eyal Kazin PhD
Towards Data Science
28 min read3 days ago

--

Life is like a box of chocolate. Generated using DALL-E

My momma always said “Life was like a box of chocolates. You never know what you’re gonna get.”

— F. Gump (fictional philosopher and entrepreneur)

This is the second article in a series on information quantification — an essential framework for data scientists. Learning to measure information unlocks powerful tools for improving statistical analyses and refining decision criteria in machine learning.

In this article we focus on entropy — a fundamental concept that quantifies “on average, how surprising is an outcome?” As a measure of uncertainty, it bridges probability theory and real-world applications, offering insights into applications from data diversity to decision-making.

We’ll start with intuitive examples, like coin tosses and roles of dice 🎲, to build a solid foundation. From there, we’ll explore entropy’s diverse applications, such as evaluating decision tree splits and quantifying…

--

--

Towards Data Science
Towards Data Science

Published in Towards Data Science

Your home for data science and AI. The world’s leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.

Eyal Kazin PhD
Eyal Kazin PhD

Written by Eyal Kazin PhD

Hi 👋 I'm Eyal. My superpower is simplifying the complex and turning data to ta-da! 🪄 DS/ML researcher and communicator. Cosmologist with ❤️ for applied stats.

Responses (4)