The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models

I’ve been trying to bust the myth about LLMs' capabilities for a while, and now it’s time to do the same for LRMs (Large Reasoning Models). Don’t get me wrong, I’m still amazed that we have systems like DeepSeek, o1, and Gemini. But as an AI researcher, it is my job not to get swayed easily by all the hype and look beyond the fluff. And today, we are breaking the myth about LRM's reasoning capabilities. Apple recently released a paper on LRM's reasoning capabilities, and the paper confirmed many of my own hypotheses. So, without further ado, let’s break this down and see this new paper in detail.

Table Of Contents

• Understanding Reasoning
• How LRMs Are Different From LLMs?
• The Illusion Of Thinking
• LRM's Performance Drops Sharply For Complex Tasks
• More Experiments
• LLM “Reasoning Traces” Or Large Mumbling Models
• Deeper Problems With LRMs
• Conclusion

Understanding Reasoning

In his 2019 paper “On the Measure of Intelligence,” François Chollet defines intelligence as “skill-acquisition efficiency,” emphasizing the importance of generalization and adaptability over mere task-specific performance.