What are the limits of ChatGPT and current natural language processing?

Taking inspiration from the century old Gödel’s completeness theorem to define the limits of natural language processing and peer into what might the next breakthroughs be…

Benchmark performance over time with initial and human performance are normalized to -1 and 0 respectively (Kiela et al., 2021).

In the last 5 years, there have been tremendous research breakthroughs in the field of natural language processing. We are starting to reach near human level accuracy in many language tasks including question answering, summarization, dialogue, translation, etc. But what are the limits of natural language processing?

Before we get to the bottom of what the limits of current AI algorithms are given the types of data and models — let’s talk about something marvelous that we do as humans. Let’s say I show this clock below. Now if you are like me, you’ve probably never seen this language before — but still, you can tell the time up to nearly 1–5 minutes accuracy.

Benchmark performance over time with initial and human performance are normalized to -1 and 0 respectively (Kiela et al., 2021).

Pause and give yourself a pat on the back! Think about how remarkable you are — you have never read or even seen Mongolian script before, yet you have no trouble…