Go back to when you were an infant, and seeing your first three, five, seven cats. How did your brain figure out what a cat was? Most explanations of this talk about how, over time, the relevant characteristics of a cat are abstracted out and a model of a cat is developed. This is also how recent AI is described; deep levels of neural nets that automagically filter out the most relevant features leading to higher and higher abstractions that more and more closely fit the “cat” category.
An upside down way of saying the same thing is that the brain is really really good at forgetting. Even before information reaches our brains, our senses have filtered most of it out — only the visible wavelengths, and only high resolution at the center of the eye, for example. However, the interesting forgetting happens in the brain itself. We need to forget about the grass, the rock, the colors, and who knows what else, in the picture to the left in order to recognize the cat. Ignoring the right details is essential.
This would lead us to think of forgetting as a real-time process; as information enters the brain, the most relevant abstractions propagate along through our neural nets, while the rest is dropped. The same, approximately, happens in deep learning. There is one major loop — input, train, repeat.
However, we have another stage of forgetting as humans — sleep and dreams. Some older research, but also some brand new stuff, supports that sleep is another way the brain actively forgets (which results, of course, in what you remember). When you are sleeping the input from your senses is limited, so the forgetting that happens during sleep must be quite different from the forgetting that happens during sense processing. What’s that difference? No-one really knows, but it’s fun to speculate. After all, if we can add this second layer of forgetting to our AI systems, perhaps they’ll leap to the next step of performance.
The most basic guess for how we forget during sleep is to assume that the brain randomly deletes (mostly recent) information. If repetition and reinforcement can create multiple instances of information, then a random deletion process would actually fit pretty well with human experience. If important information that has been replayed through your memory many times has established multiple copies (at least in the short term), then random removals will, on average, delete the information that is least important to you. Filling the holes that are created by this process requires more rounds of learning, refining the abstraction layers over time and filling out edge cases. The right balance of forgetting will actually reinforce and harden (make more resilient to noise) your learnings (i.e. limit over fitting). It could also stimulate “intuition” as you find ways to fill those holes with reasonable information. Of course, sleep and dreams may invoke a smarter algorithm than “random” forgetting, making the process more efficient.
Even more speculative, this second layer of forgetting may lead to empathy. The more you forget, the more you rely on others to understand situations, to make progress, and to survive. Group memory, which partially compensates for active forgetting, may be essential for progress. Once you rely on others for memory, you need to develop empathy for them — you become co-dependent.
In order to learn, you must forget…and by forgetting you develop empathy. A typical evolutionary tradeoff.
Science Fiction? Overly simplistic? Yes. That’s largely why I wrote the Upside Down series — to explore this aspect of forgetting, and to see if we will eventually have empathetic AI’s which co-rely on us for memory. Perhaps it’s time we add another layer of forgetting to our algorithms? It may be faster and easier to explore this thesis in software than by fulling dissecting our biology. There is actually a lot of research and experimentation going on in this area — just DuckDuckGo for “artificial intelligence active forgetting” and you’ll get a few pages of interesting reads.