Where is the Artificial Ingenuity in Deep Learning?
In a recent lecture by Demis Hassabis at the Rothschild Foundation, Hassabis explains intuition and creativity. Intuition is ‘implicit knowledge acquired through experience but not consciously expressible’. Creativity is ‘the ability to synthesize knowledge to produce a novel or original idea’.
Hassabis describes Deep Learning as an interpolative kind of creativity. AlphaGo-like systems (or methods employing reinforcement learning and self-play) have, according to him, extrapolative creativity. That is, new knowledge (i.e. new game strategies) are created by the system.
The state-of-the-art in Deep Learning is capable of a primitive kind of intuition that can support both interpolative and extrapolative creativity. To achieve the kind of inventive creativity that Hassibis seeks requires a more fundamental and basic kind of intuition. I shall use the term “ingenuity” to describe this intuitive capability that Hassabis alludes to.
It’s important to note that Hassabis’ classification of creativity isn’t discrete but rather spans a continuous spectrum. What I mean is that extrapolative creativity requires interpolative creativity. So in the case of AlphaGo-like systems, self-play and tree search are combined with a Deep Learning network to ‘imagine’ and evaluate new scenarios.
What is ingenuity and where does it originate from?
My current research program is the exploration of cognition from the starting point of understanding intuition. I’ve described intuition as that unconscious heuristic process that is developed from our experiences. It is what Daniel Kahneman describes as System 1 (i.e. Fast Thinking). What I’ve failed however to describe is that the notion of intuition also commonly associated with what is commonly recognized as ingenuity.
Alan Turing was perceptive enough to identify this specific distinction:
“Mathematical reasoning may be regarded rather schematically as the exercise of a combination of two facilities, which we may call intuition and ingenuity.”
Turing recognized that experiential learned inference and clever inventiveness are two kinds of facilities that are important in combination. What is very profound about Turing’s statement that creativity in a highly rational domain requires two facilities that are intuitive in nature.
Intuition does not only involve quickly recognizing situations, but it also includes that uncanny ability to combine ideas to develop ingenious solutions.
Aaron Sloman writes:
A good theory of mind … needs to explain the abilities of at least one sort of mind to discover and use mathematical truths about what is and is not possible.
Sloman recognizes that ingenuity exists as a different kind of intuition. According to Sloman, this is an intuition that animals like the New Caledonian crow also exhibits. So when AlphaGo decides on move 37, it does so driven by intuition but it is entirely an emergent ingenuity.
Human intuition automatically tells us what is and what is not possible. That is, we are naturally predisposed to recognize the adjacent possible. We never need to ponder whether we can survive jumping off a cliff or not. Studies of early infant learning indicate that intuitive understanding of the world is learned over time. Object permanence beginning at 2 months. Gravity, inertia and momentum understanding begins at 9 months. Shape constancy learning begins in 12 months.
A unique capability of biological creatures is the unique ability of re-purposing objects as a way to solve a problem. We don’t just recognize classifications, rather we recognize possibilities of action. This can be as simple as recognizing obstacles, to something more advanced as leveraging and inventing tools. We can recognize objects in our midst as not only obstacles but also as tools. This simple ingenuity is beyond anything a robot can do. How does this kind of cognition even work?
Interestingly enough, the previous studies in “Pattern Languages” or “Design Patterns” gives us ideas about the nature of the discovery of ingenious solutions to complex problems. In Hassabis’ lecture, he talks about Generative Query Networks (GQN) that implements a kind of imagination capability for Deep Learning. GQN reconstructs 3D renditions of the world from just 2D snapshots. To achieve ingenuity, however, imagination (or generative worlds) are required as a bare minimum requirement.
In Pattern Languages there are three elements that make them generative. A pattern language that is generative not only tells us the rules of arrangement but patterns show us how to construct these arrangements that satisfy rules (or constraints).
First, pattern languages attempt to address relationships explicitly. Typically, “We give names to things but we don’t give many names to relationships.”
Second, pattern languages and other generating systems produce effects greater than the sum of their parts.
Third, pattern languages are generating systems that contain the mechanism for their own propagation.
Christopher Alexander writes:
Chomsky’s work on generative grammar will soon be considered very limited… It does not deal with the interesting structure of language because the real structure of language lies in the relationships between words — the semantic connections. The semantic network — which connects the word “fire” with “burn,” red,” and “passion” — is the real stuff of language. Chomsky makes no attempt to deal with that and therefore, in a few years, his work will be considered primitive.
The structures of a pattern are not themselves solutions, but they generate solutions. Patterns that work this way are called generative patterns.
There are two kinds of systems that are important, a holistic system and a generative system. A holistic system is a system that has the emergent property that the sum of the parts is greater than the whole. Emergent behavior is a product of the interaction of its parts.
Now a generative system cannot be simply viewed as a single object. Rather it is a kit of tools, with rules about the way these tools may be combined to create new tools. To be able to create holistic systems one needs to discover or invent generative systems.
Artificial Ingenuity is a generative system for abstract thought. However, its core capability is the intuitive understanding of how one concept can be used as a tool to construct other concepts. Intrinsic in this kind of thought is the capability of intuitively recognizing the possibilities of recombination or reconfiguration to arrive at a new concept. Douglas Hofstadter (author of the first book ordered at Amazon) would describe this capability as analogy making (i.e. the core of cognition).
Artificial Ingenuity relates to the information discovery asymmetry that was discussed previously. In general, intuitive cognition can be framed as ‘amortized inference’. Humans develop intuitive algorithms (aka heuristics) through experiential learning. However, ingenuity requires a yet to be discovered capability. This capability exhibits continual learning that constantly improves the conceptual abstractions it generates. In conventional Deep Learning systems, the abstractions that are generated are opaque to introspection and reflection. Ingenuity perhaps requires more explicit models of analogies. These models are explicit enough that a person is able to effortlessly recognize the ‘similarity’ between two analogies. The key to this mechanism I suspect relates to our own biological need for sleep.
Further Reading
.
