How exactly did ChatGPT learn to do arithmetic?
If you’ve been following the GPT story for a while through its updates – and especially the excitement of ChatGPT and GPT-4 – you’ll probably be as stunned as we are that ChatGPT does what it does ‘simply’ through training to predict the next word.
I say ‘simply’ because of course that ‘next word’ prediction is virtually supernaturally relevant, taking into account up to the last 25,000 words of yours. Not to mention its training from terabytes of the internet.
Arithmetic?
A couple of added bonuses that not many of us had thought about before the reveal was that GPT can follow instructions (see here why or how) and . . do arithmetic.
If you’re not thinking hard, some might be saying ‘it’s a computer, of course it can do arithmetic’.
The correct answer being: yes, but GPT’s neural net algorithm only works with words and strings. The arithmetic units of its CPUs and GPUs are just working on the math of the neural network algorithm.
It doesn’t even know what a number is.
So how does GPT learn arithmetic?
Obviously it learns the rules of arithmetic like it learns any other patterns of language, logic, human common sense and knowledge:
