Comparing Logic and Math Capabilities of GPT-o1 against other LLMs (Part 3) — Logic problems

On the next parts of the series, we will delve into a number of logic puzzles. As such, they are probably more interesting, but can also pose somewhat of a challenge to the reader. I will be listing the solution to the end of each post so the inquiring mind has a chance to try and solve the problem without spoilers, and have the solution at hand as well.

Our first riddle (and the next two that will follow) comes from the book: “The Lady or the Tiger” by Raymond Smullyan a well-known mathematician and logician.

A case of Johns

Query:

There was a judicial investigation involving two twins. It was known that at least one of them was never telling the truth, but it was not known which of the two. One of them, John, had committed a crime. (John was not necessarily the one who was always lying.)
The purpose of this investigation was to uncover which of the two was John.

“Are you John?”, the judge asked the first twin.
“Yes I am.”, was the answer.
“Are you John?” the judge asked the second twin.
The second twin answered “Yes” or “No”, and by that answer the judge knew exactly who John was.
Was John the first twin or the second?

GPT-4o

Claude 3.5 Sonnet

Gemini

GPT-o1-preview

Very interestingly all the LLMs gave the same response regarding which twin was John albeit offering different explanations as to why.
What do you think, is the first twin indeed John?

Solution

Let’s assume the second twin answered “Yes” to the judge’s question.
If that was so, then the judge would have no way of differentiating between the two twins as they would have given the same answer to the same question.
So the second twin must have answered “No” to the judges question.

Scenario 1: The first twin lied and the second one told the truth.
That would lead to both not being John (the first said “Yes” but lied so he can’t be John and the second one told the truth but denied being John so he can’t be John either.)
So this scenario is not possible.

Scenario 2: The first twin told the truth and the second one lied.
That would lead to the first one being John (as he tells the truth and said “Yes”) and the second one also being John (as he lies and says “No”).
Since one of them is John this scenario is also impossible.

Scenario 3: The first twin lied and the second one lied.
In this case, the first twin is not John (as he lied and said “Yes”) and the second twin is John (as he lied and said “No”).
This scenario is possible and as it is the only one that is possible, it means that John is the second twin.

So the correct answer is that John is the second twin.

As we can see, all models failed to deduce this and got mixed up. Most of them seem to have lost it somewhere in the logic of proving the different scenarios. Interestingly enough, GPT-o1 deduces that the second twin is John, only to change it in the final conclusion without providing an explanation.