Massive Language Models Are Cheating. Should We Care?
The world of AI seems to have split into two camps recently: the die-hards and the ditchers.
The Die-hards
The hopeful camp counts members like OpenAI’s chief scientist Ilya Sutskever and Tesla’s Sr. Director of AI Andre Karpathy. They believe that some of the abilities of the best language models today are indicative of even more powerful emergent properties. Both researchers have gone as far as to lightly suggest that these models may have some inklings of consciousness. For perhaps the most dystopian and compelling take on this, check out Karpathy’s short story from the perspective of a conscious language model (related: The AI Moment).
The Ditchers
But there is another, more grim view with some traction as well. The ditchers not only say that models aren’t conscious (they really went after Sutskever for his take), some also say that they never can be. There are a lot of reasons that people hold this view. These ideas are tricky, in my experience, because most of them make sense at first, but their conclusions are ultimately a matter of opinion.
Take the most famous such idea, the Chinese Room Experiment. It basically purports that if an english speaker went into a room with an English-Chinese dictionary, she could convince people writing to her from the outside that she spoke Chinese. The argument is that the English-speaker can simulate language understanding without actually having it. It follows that ML models can learn rules to simulate consciousness without actually being conscious. But at the end of the day, even if the person in the room doesn’t understand Chinese, the question of whether the room (containing both the person and the dictionary) can be said to understand Chinese is a matter of personal opinion.
To see this, imagine that the brains of everyone you know are inhabited by little aliens who only speak an alien language. They translate their thoughts to English before speaking to you. Would you doubt if everyone you knew understood English? Even after discovering the aliens? A matter of personal opinion, in my book.
Model’s Are Cheating
The first time I ever felt the urge to ditch Karpathy and friends to become a pessimist was when I read this thread by Alex Tamkin, a researcher at Stanford who mentored me for some time. In it, he explains briefly how it may be the case that GPT style models are simply memorizing training data very well, rather than learning generalizable patterns. At face value, this seems to blow the murmurs of consciousness out of the water. How could a model be consciousness if it’s really just remembering the right thing at the right time?
But then, of course, aren’t we all just remembering the right thing at the right time? Think about exactly how many of the phrases you say you have heard before. One paper cited by Tamkin implies that the frequency of arithmetic expressions in training data for a model is predictive of how accurately the model can work with the expressions. Another neural system this is true for is my little cousin Isabella, who was able to do consistently better on her multiplication tables after she practiced them and received feedback. Filmmaker Jim Jarmusch put it best:
“Nothing is original. Steal from anywhere that resonates with inspiration or fuels your imagination. Devour old films, new films, music, books, paintings, photographs, poems, dreams, random conversations, architecture, bridges, street signs, trees, clouds, bodies of water, light and shadows. Select only things to steal from that speak directly to your soul. If you do this, your work (and theft) will be authentic. Authenticity is invaluable; originality is non-existent.”
Just like the Chinese room experiment, whether or not you would ever call an advanced memory-resurfacer conscious seems to be a matter of personal opinion. Could this sifting and selecting of relevant text really be the origin of thought? In the case that it is, perhaps the ideas that fire through our heads are nothing more than the flipping of pages in the book of our own experiences and exposures.
