model.fit() part II
Memory
Memory is our nemesis. Ever felt something was at the tip of your tongue, but you just can’t dig it out of your brain? Or why the brain throws random memories at you when you’re engrossed in something terribly important? Why do you suddenly break into the tune of a song you may have heard only once in your childhood? Why do you sometimes forget what you forgot?
We grow up with the habit of writing on paper, or on screens, and writing seems to be the only mode of storing information. When we need to store a number ‘8’, we can not think of any other means other than writing the symbol ‘8’ somewhere in a piece of paper, a notebook, or a tab. If the brain worked in this fashion, it would have written something like this
‘8’ = ‘eight’
There would have been a specific ‘eight’ neuron for recognising ‘8’ from visual input, and a ‘nine’ neuron for ‘9’. Which is all very well. Except that if we classify the world in such a manner, the sheer number of different objects in the world would soon overpower the 100 billions neurons of our brain.
There are neurons in our visual cortex which can detect geometric shapes like lines, and those which detect edges. You can describe a ‘phone’ as something rectangular, a ‘donut’ as two concentric circles, a cup as a oval-with-an-ellipse-with-a-semicircular handle (or some equally complicated description). Such feature extraction techniques work for a limited set of objects; but the real world isn’t geometric, and far too big for such simplistic models.
Clearly, having once seen the principle of what is sometimes called feature extraction, we could go on like that for ever. We can join line detectors together to make detectors for squares, for numbers, for letters of the
Russian alphabet, for teacups and so on ad infinitum. Is there any limit to this? Is there a special cell for recognizing every single thing we see? It is not at all clear that even with the countless numbers of neurons at our disposal
we could actually do something like that. For teacups, just possibly: but for porcelain teacups, for eighteenth-century French porcelain teacups? It doesn’t sound very likely, if only because the number of cells required would soon
exceed even the million million that we are provided with. So how is it done?
— Roger Carpenter, Neurophysiology
It must be obvious to the reader that the brain does not quite work in this manner. There’s nowhere to ‘write’ anything inside the brain, and there is not a particular neuron for each object in the real world. Instead, information is stored much subtly, in a manner only evolution could have thought of.
Do you remember the first time you saw the symbol ‘8’? There are millions of neuron in your visual cortex (the back of your head), but only a few of them would get activated, or ‘fire’, when you saw ‘8’.
A Canadian psychologist, Donald Hebb, proposed in 1949 that all these neurons that ‘fire together’ on a certain stimulus, __repeatedly__ (i.e. everytime you see ‘8’) would also develop more numerous and stronger synapses¹, i.e. they will ‘wire together’. The hypothesis was later proved in the works of Eric Kandel, and discover of specialised synaptic receptors in the brain.²
This kind of storage, i.e. distributing information across a network has the advantage that one can also recognise ‘9’ by the same network, albeit by activating a different set of neurons.
What about ‘8’ written differently, such as in a different font
8The beauty of the Hebbian model is that once tight synapses are formed, any input will gravitate towards a few selected outputs only. Once the child recognises ‘8’, any kind of ‘8’, anything remotely resembling ‘8’ would be classified as ‘8’ in this model. The similarities between the two ‘eight’s, i.e. their common eight-ness would produce a more or less similar pattern of activation in the network.
Infact, by distributing memory across myriads of neurons, the brain has achieved an almost infinite capacity for storage; it never complains that it is ‘out of memory’. One can always find one more pattern of activation (i.e. green dots) to store another piece of information. The simplistic network presented in the diagram, with binary activation (i.e. a neuron can only either be be fully green or fully white) and 16,12 and 10 neurons in three consecutive layers, respectively, can store 16 x 12 x 10 = 1920 pieces of information. The brain has 100 billion neurons, which can be fully or partially activated (i.e. they are not binary, but their activation can be an intermediate number like 0.612); there seems to be no limit to how much information this dense and huge meshwork of neurons can store.
The part where this gets really interesting is middle layer of this network (the ‘hidden’ layer). Almost the entire brain of a rat is dedicated to either a sensory or motor function, i.e. they are the two ‘ends’ of the network. In humans, however, the large majority of brain is neither sensory nor motor, but ‘association areas’ (i.e. hidden layers).
One can imagine packets of information traversing to and fro through these layers, and suddenly apacket ‘leaks’ onto a different route, and you are reminded of that Ghazal by Pankaj Udhas you heard on a train compartment on someone else’s phone, maybe years ago. Or a packet might be dropped entirely, once in a while. And sometimes, the information of a lost packet might surface: you remember that you have forgot something before!
…
- Hebb DO (1949). Organization of Behavior: a Neuropsychological Theory. New York: John Wiley.
- Carpenter H. Neurophysiology 5e (2013); CRC Press:259
