The aha of Monty Hall
This great geeky gotcha mostly ends not with an aha! moment of understanding, but anti-climatic calculations, scared wonder at the difficulties of probability, or in the case of Brooklyn Nine Nine, Freud. Here’s an attempt at finding the aha!.
TL; DR: the host is leaking to you
The Monty Hall Problem is so fun and counter-intuitive that it has appeared in the likes of quality novels, the film 21 or even the delicious goofball TV series Brooklyn Nine Nine.
However, a recent chat with a friend showed me that, despite such popularity, it is not easy to find many who are at peace with it.
There are many reasons. The standard solution is formulaic. Even if one understood how to derive it, there’s still a difficult step further to understanding where the gotcha comes from. Most attempts at intuitive explanations, while stirring up the mind and conversations, still miss that aha explanation.
Let me try to find the aha.
The Problem
Here’s the Brooklyn Nine Nine version of the problem:
Imagine you’re on a game show. There are three doors, behind one of which is a car. You pick a door. The host, who knows where the car is, opens a different door, showing you there’s nothing behind it. Now the host asks if you’d like to choose the other unopened door. Should you do it?
Should you stick to your original door or switch to the other unopened door?
Classic Wrong Answer: sticking or switching has equal odds
The host opened a door, but this does not suddenly move the car from one door to another. The unopened doors remain equivalent and have equal odds of having the car. So switching wont improve the odds.
However, we conclusively know that this is the wrong answer. This is not just from theory but computer simulations of the game and even real world experiments.
Right Answer: switching has double the odds
The right answer is, you should switch. Your chances double if you do so. Again, we know this conclusively from several sources of real evidence.
Explanations
Since the question is about probabilities, the classic solution involves simple probability calculations. It is neat and general. Yet, people often feel it is missing something, not to say boring, a case of Shut Up and Calculate. Why did one door suddenly have more odds?
Brooklyn Nine Nine’s or 21’s solutions are not very helpful either. The former briefly says “probability locks in when you make the choice” before settling the posing of the problem as a simple case of not having sex.
my answer’s based on statistics. Based on variable change. when I was originally asked to choose a door, I had a 33.3% chance of choosing right. But after he opens one of the doors and then re-offers me the choice, it’s now 66.7% if I choose to switch.
This merely lead to a lot of Google searches about “variable change” and why it leads to the solution.
There are better intuitive explanations from statisticians. One goes like this. Consider the game with 100 doors instead of 3. You first choose a door, the host opens 98 other doors without a car and gives you with an option to switch to the last door standing. Wouldn't you intuitively feel that you should switch?
Here’s another. Consider the game with 3 doors. However, after your initial choice and before the host opened any door, you are given a choice to switch to both the other unopened doors. If either of those had a car behind it, you win.
In this changed game, it is obvious that you should switch. If the host now showed that one of those doors did not have the car, you are still better off keeping to the switch. Voila?
(Both 21’s and Brooklyn Nine Nine’s solutions are versions of this)
Both are nice attempts. Yet, while they make one get more intuition about the counter-intuition, they still don't pinpoint to a clear aha. There’s some vagueness and a sense of having to trust axiomatic assertions.
And thus for many it is still not obvious to switch: they vacillate between the gut feeling to switch generated by the above examples, and the idea that both remaining doors are still equivalent and the cars did not move.
I am often asked why people tend to find probability a difficult and unintuitive idea, and I reply that… it is because probability really is a difficult and unintuitive idea.
— Sir David Spiegelhalter, President of the Royal Statistical Society
The aha of Monty Hall: the host is leaking to you
The confusions arise because we tend to ignore one crucial aspect: the host’s action of opening a door.
After your initial choice, the host’s choice from the other two unopened doors could not be done with a coin toss.
A coin toss here means the choice between the two doors is done in a purely random way, with equal odds for both.
Instead, when the host opened a door, he had to be first sure that there’s no car behind it. If he used a coin toss to select a door, there’s a chance that he could have opened the door with the car and the game will collapse.
So, he will look behind both the doors for the car, and:
- If neither has a car, he will choose one with a coin toss
- If one of them has a car, he is forced to choose the other
Thus, on average, in 2/3 times the game is played, his choice leaks to us the fact that the door he left standing has the car.
In the case of 100 doors, 99/100 times the host pinpoints us to the door with the car!
He does this by opening everything else but the one with the car and your initial choice. Only 1/100 times your initial choice will have the car!
His pinpointing leak is what makes the two unopened doors not equivalent.
There’s your aha.
Thus, on average, it is no longer an equal probability coin toss between switching and sticking. The formulaic calculations do that averaging behind the scenes.
We get all this from the structure of the game.
To me, it’s a scandal that many of the well meaning attempts at intuitive explanations leave this out.
A Better Way
Even after my decades as a statistician, when asked a basic school question using probability, I have to go away, sit in silence with a pen and paper, try it a few different ways, and finally announce what I hope is the correct answer. — Sir David Spiegelhalter
Conceptual aha is still perhaps prone to mental vacillations. However, there’s a final solution to settle this for once and for all: the method of implementing it in code.
The moment you try to write down the problem in code, the abstract has to give way to the concrete. Your mind’s weaknesses regarding thinking about many abstract possibilities at a time gives way to a structured understanding of them. You will not just get a solution, but clarity and explanation.
It will easily dawn on you why switching has higher odds, since you have to implement the host’s process of choosing!
An implementation
Let’s create 100000 trials of the game, each with doors 1, 2 and 3. And then put cars behind a door for each of them.
For simplicity, let’s say your initial choice is always 1, and the choice is to stick to 1, or switch to the other unopened door after the host opens one of 2 or 3.
The implementation in python is very simple.
Let’s plot a histogram to see if cars are put behind doors correctly, i.e., with equal odds:
Now, let’s get into important part.
It is difficult to convey the feeling of this, since aha moments are so personal, so trust me with this: Writing the following snippet, by oneself, really clarifies the problem.
If you are still unconvinced, try writing your own implementation from scratch :-)
Let’s play the games.
Finally, the moment of truth:
Voila. There you have the twice higher chances for switching instead of sticking. And in this case, the coder can be happier: you made it yourself at home, with your own hands! :-)
The same approach — computational than mathematical or formulaic , code instead of math — has caused a revolution in Statistics and Machine Learning. For a brief introduction, check out “Statistics for hackers” by the wonderful Jake Vanderplas (and think of saying good bye to the likes of t-tests).
Postscript
The postscript is from the novelist Mark Haddon. His famous novel covered the Monty Hall problem in some detail. That’s where I heard about it for the first time.
I get many letters explaining, at great length, why I’ve got it wrong... The Monty Hall Problem is famous precisely because the correct answer is so infuriatingly counter-intuitive.
The irony is that if you play the game (all you need is three squares on a piece of paper, a pencilled cross and a dice) it becomes rapidly obvious... But there are many very intelligent people who believe that thinking about something is superior to doing it…
