Probability is in the Eye of the Beholder… Probably

We can talk about probability with a shared understanding. But what does it mean to say the odds of rolling a 4 is 1/6? The exact meaning is a little difficult to say. In this blog, I’ll talk about the formal interpretation of probability, why it matters, and why interpreting is difficult.

A Riddle

I start the blog with a riddle:

It’s American Football season. The Orcas are scheduled to play the 99ers on Sunday. The Orcas have a superstar quarterback, Joe Football. If Joe Football plays on Sunday, then the Orcas have a 90% chance of winning. But if Joe sits out due to injury, then the Orcas only have a 20% chance of winning. On the Monday before the game, Joe decides that he’ll work out extra hard this week. This decision causes him to injure his shoulder on Tuesday. But he doesn’t realize on Tuesday that he suffered an injury. When the pain persists on Wednesday morning, he sees the team doctor in the afternoon. The doctor analyzes some X-rays Wednesday evening and tells the coach about the injury on Thursday. The coach decides Thursday night to have Joe Football sit out due to the injury. He tells Joe on Friday morning, and the media on Friday night. By Saturday afternoon the Las Vegas odds reflect the new 20% chance of winning. The riddle is this: The odds changed from 90% to 20%, but when?

Riddles are never straightforward, so maybe the setup is misleading. Were the odds ever actually 90%? Because Joe didn’t play on Sunday, is it meaningful to say that the he was “going to play”? Or was Joe’s injury inevitable? This inevitability argument is interesting, but it doesn’t go far enough. The Orcas eventually lose on Sunday; does that mean their odds were always 0%? Could a super-powerful computer with enough information predict that Joe would be injured, and that this injury would cause the team to lose? Could that powerful computer be >99% confident? Let’s say yes, at least theoretically. So then where did 20% come from? That’s the trick.

The last part of the riddle says that the Las Vegas odds changed to reflect a 20% chance of victory, so maybe the riddle means that Las Vegas says the Orcas have a 20% chance of winning. But this 20% belongs to those casinos. It describes how those casinos have set their odds (usually a reflection of their customers’ collective betting behavior). This is not an independent, universal property of the game that the casinos merely discovered.

In fact no such universal probability exists. As the title says, probability is in the eye of the beholder; it exists within each person or model or group of people. Probability is an expression of an expectation, and so it must be somebody’s expectation. In the riddle, Joe, the doctor, and the coach may have each updated their expectations at different points throughout the week. Their expectations don’t need to agree with Las Vegas odds. In fact the casinos often don’t perfectly agree with each other. The trick of the riddle is that it pretends that probability can be defined objectively without reference to an expectation-holder. The riddle is tricky because we have a bad habit of talking about probability as though it is universal.

How we define probability.

The big problem posed by the riddle is that probability isn’t clearly defined. We may not agree on what we mean by probability, even though most of us are fairly comfortable mentioning probabilities when talking about sports, politics, weather, etc.

Probability may be defined using a “frequentist” interpretation. By this interpretation, a probability(rolling a 4) = 1/6 means that if you roll a die N times, you should get a 4 about N/6 times. This is inadequate for a football game, because the game only gets played once.

An alternative definition of probability uses a “Bayesian” interpretation. This interpretation says that probability measures expectations. This is how we’ve been talking about probability this whole blog. But what exactly does “expectation” mean? This is usually explained: Expectation measures what it’d take to get you to bet. If I offered to pay you X>$6 every time you roll a 4, you’d gladly pay $1 to play. But if I offered to pay you X<$6 every time you roll a 4, you wouldn’t pay $1 to play. So you expect to roll a 4 with probability 1/6.

It’s common to explain expectation with betting; this is not just an artifact of my love for gambling. But measuring expectation as a willingness to bet isn’t perfect. We’ll talk about shortcomings in the last section.

Why it matters.

It’s important for all of us to understand that probability is a measure of expectations.

One reason that understanding is important is that it helps us avoid over-confidence. If you’re told the probability of something is X, you can ask, “says who or what?” Some probabilities are more informed than others; you can decide how much you want to believe that probability. I believe a dice roll model, but I’m skeptical of a stock-movement model.

It’s also important to understand that everyone has expectations, even in the face of uncertainty. If you ask me, “What are the odds of raining in Seattle tomorrow?” I don’t have a good idea — I don’t think much about Seattle, and don’t follow weather closely. But I know that Seattle rains a bunch. But on the other hand, most cities don’t rain most days. Weighing these, I guess I’d say 60%. This isn’t a good model, but it’s my model. It’s uninformed and it’s biased. If it mattered, I’d do some research to form a better opinion. But the point is: My probability exists even before I become informed.

We sometimes make the mistake of thinking precise odds imply high confidence. But a probability, even an uncertain one, is meaningful and useful. Giving uncertain probabilities is a habit of careful thinkers: For example, Tetlock and Gardner have a relevant chapter called “Keeping Score” in their book “Super Forecasters.” There they emphasize the importance of experts expressing their expectations as probabilities when they report National Intelligence Estimates.

When somebody asks you “What are the odds?”, the answer “I don’t know” isn’t correct. Because “the odds” live in the eye of the beholder, so the person really wants to know what are your expectations? You know what your expectations are. If you’re completely uninformed, then 50–50 is an appropriate answer. If you’re asked the odds that The Seabirds beat the The Rushers, and you haven’t heard of these teams, then 50–50 is right for you. But usually you’re asked about things you know. If your boss asks the probability that a project completes this quarter, there may be a lot of uncertainty (other teams, sick days, etc.), but you have informed expectations.

One caveat is that “What are the odds?” is often shorthand for “What is the consensus expectation?” If somebody asks you the odds of a Roulette wheel coming up 42, and you don’t know what a Roulette wheel looks like, then “I don’t know” actually is more appropriate than saying “6 percent.” You need to use context and basic human-ness to decide how to talk to people; I can’t really help here. But speaking of consensus expectations…

Objective or Subjective

So far we’ve insisted that probability lives in the eye of the beholder. This is called the subjective variant of the Bayesian interpretation of probability. An objective variant also exists. This uses the same expectation interpretation, but relies on some shared understanding — it says that probability represents a “reasonable expectation.” So instead of saying what a given person would bet, it says what an ideal rational actor would bet (whatever that means).

If you try to compute reasonable expectation for the riddle above, you might ask “reasonable to whom?” Because each person should reasonably have different expectations, it seems like the objective variant doesn’t work. But you can maybe fix this by saying, “for a fixed set of data, there’s a universal (objective) reasonable probability.” In the riddle, Joe Football might be the first to know about his injury, but he might say that this makes his team more likely to win. We want to be able to say that he’s wrong in some objective way.

The objective Bayesian interpretation is useful. Different modelers using different approaches usually get similar answers (when working with the same data). It’s useful to understand that these modelers are both approximating some objectively “correct” probability. Even if we think of the world as deterministic, there’s no way to juice a 100% prediction out of a finite set of data.

Objective Bayesian probability may make sense for fixed data. But usually it’s hard to distinguish modeling from data supplementing. Different modelers could lookup injury rumors, but if / how to use that data will vary between them. If a modeler could look that data up, but chooses to not, it’s not clear if we should count that data in the fixed data. This is a little blurry, and this is okay. Objective vs. subjective Bayesian vs. frequentist are different frameworks of understanding that are more or less helpful in different situations.

Some nuance

Above we said that expectation measures how much you would be willing to bet. This isn’t always true though. There are lots of non-math reasons to not gamble (social, moral) or to gamble at an expected loss (entertainment, insurance). So we’ll refine “expectation” to mean “how much a profit-maximizing person would be willing to bet.”

This still fails to account for some cases: We said that if you don’t know about sports, you may say there’s a 50–50 chance the Seabirds would win. Yet, it’s rational (profit-maximizing) for you to refuse to bet on the Seabirds, even if you were offered favorable 2-to-1 odds. This is because whoever is offering you that bet probably knows more about sports than you.

I like my 60% estimate of Seattle rain, but as soon as somebody offers to bet against me, I’ll assume they know more than I do. In fact, willingness to bet is often an indicator of informedness. For example, most people believe that a Roulette wheel that landed on black with its last ten spins is then more likely to land on red, but few people are willing to bet money on this belief.

It’s clear that there’s another dimension of expectation, which the single probability number doesn’t capture. “50/50” when you’ve never heard of the teams means something much different than when a professional sports better says “50/50”. We call this other dimension “informedness”. There are different ways to characterize informedness, but the implication is the same: If two rational people make a bet, the more informed person has a higher expected profit. “Expected profit” refers to a probability. But if informedness is a probability of correctness, does informedness also live in the eye of the beholder? Perhaps it’s up to you to judge my informedness and visa versa. This does explain why two rational people can both expect to win a bet against the other.

There’s another problem. We understand probability as an expectation, and expectation as roughly a willingness to bet. But we must understand informedness to get a full picture of expectation. The problem is that we’re trying to define informedness, itself, in terms of a probability. This is circular. There’s not an easy way out; I’ll leave it to the reader to think on this. We’ve come a long way in terms of understanding probability, but the theoretical underpinnings remain difficult.