Member-only story
LLMs Can’t Learn Maths & Reasoning, Finally Proved!
The age-old question regarding LLMs: Do large language models (LLMs) solve reasoning tasks by learning robust generalizable algorithms, or do they memorize training data?
To investigate this question, recently a paper used arithmetic reasoning as a representative task. Using causal analysis, they identified a subset of the model (a circuit) that explains most of the model’s behavior for basic arithmetic logic and examined its functionality. Now we finally have the answer to how LLMs solve maths and reasoning tasks.
So, without further ado, let’s break down these new papers on Algorithmic reasoning.
Table Of Contents
- Defining Reasoning
- Types Of Reasoning
- Understanding Heuristics
- Breaking Down Black-Box AI Internals
- Mathematical Circuits
- Understanding Circuits In More Details
- Conclusion
Defining Reasoning
In his 2019 paper “On the Measure of Intelligence,” François Chollet defines intelligence as “skill-acquisition efficiency,” emphasizing the importance of generalization and adaptability over mere task-specific performance.