Information Theory

Chelsea Lawson
Nov 16, 2017 · 4 min read

“Information Theory” is the short name given to Claude Shannon’s Mathematical Theory of Communication, a 1948 paper that laid the groundwork for the information age.

It is not as much about information as it is about the transmission of information. That is, communication.

The theory has had enormous implications on modern society, and there are still limitless applications to ponder and pursue. For now though, I’ll simply explain three of the main breakthroughs and hope you find them as cool-in-themselves as I do:

  1. The components of communication
  2. How to communicate a greater quantity at a faster speed
  3. How to transcend noise and communicate accurately

1. A Unified Framework of Communication

The first part of Information Theory says that all communication — any message across any medium — fits into the following framework:

The message starts from a source, such as a human or a machine. It gets expressed via a transmitter, such as a letter or a tweet or your vocal chords. It then travels over a channel, like airwaves, where it is subject to noise. It hits a receiver, like someone’s mailbox or twitter feed. Lastly, it arrives (or maybe doesn’t) at a destination, such as someone’s brain.

Landlines are easy to visualize. Phones act as the transmitter and receiver, and the cable is the channel.

Next, Shannon posited that in addition to a common framework for communication, there is also a common thing that is transmitted when you communicate. He called this thing “information.”

According to Shannon’s definition, something contains information if it tells you something new. Its units are measured in “binary digits” (0 or 1), better known by the condensed form “bits.”

A coin toss makes for the simplest example. If I flip a fair coin, the result can be either heads or tails (0 or 1), so the outcome of the coin toss contains one bit of information. A trick coin that has heads on both sides, on the other hand, contains no information because it can’t possibly tell me anything new.*

No new information

A message (or piece of a message) that doesn’t tell you anything new is called redundancy. For example, when someone says “every single one…,” the word “single” is redundant. You can drop it and still retain the same information. Shannon estimated that the English language is 50% redundancy.

2. How to Communicate More, Faster

Being able to quantify information is significant because it allows us to compress messages by squeezing out redundancy.

Continuing with the example of the English language, if you know that “u” always follows the letter “q”, then you can drop the “u” before it enters the channel and add it back in when it gets to the other side.


This is what led to the Information Age — communicating more and more with fewer and fewer bits.

3. Maintaining Accuracy

However, as with any time one tries to increase efficiency, there is a balance in the form of quality and effectiveness. If I strip out all the redundancy in my words when I speak to you, it’s going to be harder to catch everything and the likelihood that you misunderstand the message will increase.

Further, we must assume some level of noise/interference in the communication channel. Noise can come in many forms. In the case of a phone call, maybe it’s physical background noise, or distractions the person on the other line faces.

So how do we combat noise and mitigate the risk of communication failure? By adding redundancy back in!

Indeed, this is why languages have redundancy in the first place!

Ain’t Nobody Got Time for Math

Shannon is often compared to Einstein, and I will draw an analogy here as well.

Information Theory, like E = mc² , was earth-shattering mainly because the math checked out and allowed engineers to put the concepts into practice. If Einstein just said the concept that energy and mass are two sides of the same coin, it would not have led to a Nobel prize.

But it is the concepts that I find so interesting, and that allow us to extend the theory to other disciplines and applications.

I am looking to apply Information Theory to self-improvement/self-actualization by conceptualizing communication from your higher self to your current self.** I know… it’s heady. I’m currently writing a book to fully expand on this, but a few thoughts to start:

  1. We can leverage Shannon’s communication framework by making sure that all the pieces are in place. In particular, we need to work on the receiving end (when you realize insights, where do they go?). Write things down!
  2. We can use tracking to compress experiences and feelings. (dashboard example)
  3. We can add in redundancy by reviewing our goals, tasks, insights, and data in a systematic way.

Please share your thoughts in the comments and consider supporting me on Patreon!

*Note that information in Shannon’s theory does NOT equal meaning. Jibberish nonsense, for example, IS information because the person hearing it would not have known it before. Same with a fair coin toss.

**We can apply the same concept to organizations too, but they are more complex to think about.

Ain’t Nobody Got Time For That!

Explaining things visually and in plain English.

Chelsea Lawson

Written by

One cannot fix one's eyes on the commonest natural production without finding food for a rambling fancy.

Ain’t Nobody Got Time For That!

Explaining things visually and in plain English.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade