What Learning Systems do Intelligent Agents Need? Reviews from DeepMind (1)
Complementary Learning Systems Part 1: slow and fast learning systems
Memory is the data or information retrieved when an intelligent agent needs it.The ability to retrieve and store memory is one of the features of intelligent agents. With the theoy of Complemnetary Learning Systems (CLS), an intelligent agent implements 2 learning systems— one in the neocortex for slow learning, one in the hippocampus for fast learning. Learning in an deep artificial neural netowrk can be considered as slow learning in the neocortex. Training a deep neural network is data-inefficient, even a deep neural network is poweful. A Release from DeepMind reviewed CLS and proposed that there may be insight from this theory to address this problem.
Long-term memory
Declarative (or explicit) memories are memories that can be inspected and recalled consciously, while procedural (or implicit) memories are typically unable to consciously recall them.
Explicit memory can be sub-classed into episodic memory and semantic memory.
Episodic memory is the memory of events (times, places, associated emotions, and other contextual knowledge, i.e. who, what, where, why) that can be explicitly stated.
For example, “specific memory of petting a particular cat”.
Semantic memory is the memory of general knowledge (facts, ideas, meaning, and concepts) that we have accumulated throughout our lives.
For example, “what a cat is”
Episodic memory is about experiences and specific events that occur during our lives, while semantic memory is about general knowledge.
Complementary Learning System (CLS)
“Effective learning requires two complementary systems: 1. located in the neocortex, serves as the basis for the gradual acquisition of structured knowledge about the environment; 2. centered on the hippocampus, allows rapid learning of the specifics of individual items and experiences”[1]
In the CLS theory, the above two types of memory play important roles in 2 different learning systems.
Slow learning — gradual acquisition of structured, generalized knowledge
The first kind of learning system implemented in the neocortex, in which structured knowledge (concepts, general knowledge) is gradually established by slowly changing the connections within this area.
We can observe this kind of slow learning in training a multi-layered artificial neural network. By showing a multi-layered neural network millions of images of different objects with labels (e.g. cars, airplanes, man, woman, dog, cat …). The connection weights in the network are updated gradually. After training, each layer becomes a feature extractor that can produce a representation of this abstract feature from its inputs.
For example, the hair, the eyes, the mouth, the ears, can be extracted as activation patterns from the input of the following image and the network is able to correctly classify that this image is a human face…
Abstract, generalizable knowledge — semantic memory, is stored in the connection weights in the neural network after training. Semantic memory is robust and powerful, but it does take costs. The acquisition of semantic memory can only be slow.
First, each experience represents only a single sample from the environment. The update signal from a single sample is expected to be erroneous due to the incompleteness of information.
For example, we know that there are many types of cats and the concept of ‘cats’ should be robust to variations of the appearance of different breeds of cats. The updates on connection weights produced by inputting an image of a particular type of cat cannot be too large since this image of the cat cannot represent all types of cats. Only aggregating small updates from a large collection of samples can produce more accurate updates.
Second, the connections are inter-dependent. The optimal adjustment of each connection depends on the values of all of the other connections. During the initial phase of learning, most of the connection weights are not optimal. The signal specifying how to change connection weights to optimize the representation will be both noise and weak. This slows the initial learning.
Fast learning — quick adaption to item-specific events
Instead of slowly changing the connection weights in the neocortex, through exposing to thousands of samples, the brain implements a different learning system in the hippocampus. This learning system supports rapid and relatively individuated storage of information about individual items. This ability is crucial to the initial phase of learning.
Let’s consider the classic game Ms. Pac-man!
You can try to play this game with the following link!
After you have played for a while, you would find that it is very dangerous to leave your little Pac-man near the ghosts. You will lose 1 life if the Pac-man is eaten by a ghost. Even if you have played Ms. Pac-man before, you will be able to know that you should avoid the ghosts immediately after the first encountering of the ghost.
The CLS suggests that the initial storage of an event is in the subregions of the hippocampus. This storage helps us to quickly exploit this experience that ‘eaten by a ghost’ results in a large negative reward.
The subregions of the hippocampus (CA3, DG) are thought to account for pattern separation and pattern completion. Pattern separation enables us to distinguish similar events and pattern completion enables us to retrieve a particular memory from only partial information.
You can find a more detailed description of the mechanism inside the hippocampal area after the projection from the neocortex.
The activation pattern in the hippocampal area has larger sparsity. That means that the activation patterns within this area encode events with less overlapping (distinct events). The episodic memories live in the hippocampus.
Fast and slow learning — a general framework for building artificial intelligence
The fast and slow learning system is not completely independent. They work together to support both efficient and generalizable learning. Semantic memory is generalizable knowledge. An intelligent agent can model the complexity of the environment with it (e.g concepts in physics). However, it takes really long time for an agent to capture this generalizable knowledge. An intelligent agent must expose to a tremendous amount of experiences before it really understands something robust, generalizable.
Episodic memory is crucial for situations that you need quick adaption on some tasks. We can see from the above example (Ms. Pac-man), that realizing the event ‘The Pac-man eaten by a ghost causes a large negative reward’ is crucial for achieving higher scores quicker. Individual events can be stored and retrieved from the hippocampus. The agent can void the actions that lead to devastating results. This may be the key to build a data-efficient artificial intelligence.
One interesting thing is that the retrieval and the storage of episodic memory depend on the activation pattern projected from the neocortex. The activation pattern in the neocortex depends on the connection weights within this area. The connection weights are tuned by the slow, interleaved learning. Therefore, the fast-learning system actually relies on the slow-learning system. This is the tricky part. They are complementing each other.
We will continue to describe this complementary system in the next few articles.
