The Bitter Lesson
What direction to avoid in the field of neural symbolic AI?
Bitter Lesson 1.0
In a blog post by Dr. Richard Sutton in 2019, he coined the term “the bitter lesson” to warn against the temptation of relying on human knowledge to design general-purpose agents/models. According to Sutton, a more efficient search and learning algorithm that scales with computation will always win in the long run.
This perspective is supported by the 70-year history of symbolicist methods losing to connectionist models. Every time researchers have resorted to human knowledge-based solutions, such as expert systems and hand-crafted classifiers, they have failed miserably.
Based on Dr. Sutton’s words: we need AI agents that can discover like humans, not ones that simply contain what we have already discovered.
This point is especially important for the field of neural symbolic AI. Bringing in old-world AI designs into current neural networks may seem appealing, but it is ultimately irrelevant if it goes against the trend of leveraging computation.
Bitter Lesson 2.0?
Karol Hausman, a staff research scientist at Google Brain, has named the robotics version of this problem “the bitter lesson 2.0.” He predicts that the new inevitable trend is to leverage foundation models that are getting better at understanding the human world in a machine-readable form and providing human-like assistance efficiently.
For the field of neural symbolic AI, this difference could mean life or death. What should we keep in mind to avoid standing in the way of foundation models? Which section of the NeSy methods, an overarching field for now, will survive the marching of these large language models that can already carry out logical reasoning within an acceptable margin?
I strongly suggest reading Dr. Sutton’s blog to gain a deeper understanding of the bitter lesson. Let us know what you think.
