Sparse Hierarchical Distributed Invariant Representations

Looking into Neural Code for better AIs

Keno Leon
Keno Leon
Sep 25, 2020 · 8 min read

How does your brain represent the environment ? The partial answer is the admittedly complex title which we will explore in this article.

Representation is a hard problem for both Neuroscience and AI and a good explanation of these terms I think is helpful if we are to build better AIs, this is my attempt.


A representation is simply that, the internal description of something, a thing, an idea or even a thought or feeling in general, but alas these later ones are very high in the hierarchy of representations and we don’t yet know exactly how they are formed in biological brains. A simpler representation is your perception of a physical thing out there, an apple for instance.

And how do we know they are representations ? We don’t definitely know, what with everyone perceiving the environment in a slightly different way, but we have some agreement and if I ask you what this is:

You ( and most people ) will surely answer a banana, so it follows that you had some form of stored representation of it since you recognized it.


Unlike a folder or book you store somewhere unique and specific like a bookshelf our brains seem to store things in a sparse manner, so you would break down the book and store the pieces in different places, a page here a page there, the cover over there. Memory at this level can be understood as a series of changing pointers, directions as to where you stored each element.

The Nodes (Circles) represent the stimulus (book) and the sparse elements, the connections or pointers which bind this representation.

Vision is a good example since there is good research related to how we store images and movement, let’s say you want to store a symbol you have never seen:

Rather than store the image as a whole, the image is broken down into parts (mainly via retinotopic maps/receptive fields), so for instance this is a simplified view of the encoding of the previous symbol:

Here we have 4 different visual maps ( the blocks ), the first 2 code for lines at a certain angle, the other 2 for positive and negative space representations, in the brain the blocks are thought to be groups of neurons. How we later integrate these sparse elements into a cohesive whole is not yet known and it's sometimes called the integration problem. Intermediate, temporal and recursive maps could explain them, but that's for another article.


A more complex shape and meaning requires the combination of several maps and their resulting simple shapes into complex objects*. Rich and complex perceptions seem to be stored in a hierarchical manner, this means that lower/simpler representations are recruited to form them:

Here we show only a couple of visual maps that when joined in the next hierarchical level form a word and then in the next level a representation of food.
It is unknown at what level hierarchies appear or if they are the only way to encode complexity but neurons seem to code ever more explicit things as you move away from primary sensory areas, the representation of a familiar thing like a face, place or in this case food can be achieved by a few neurons, but as mentioned you need the whole brain and systems along with experience to make sense of them. We share the rough distribution of early sensory systems and bigger cognitive blocks, but the content at some point becomes particular to the individual. (ie we might share a place(s) where we store the detail of faces of our family members, but they all have different content).
*Here objects can be of a different nature than visual stimuli, hearing speech for instance combines phonemes, time features and other lower maps into words we derive meaning from :


It would be terrifying and not very practical to lose the memory of your nana if a few neurons went missing, so it is convenient to have representations distributed across multiple connections and areas of your brain. In how many areas and how exactly are these distributed presentations stored is still not fully understood but basic sensory representations are a good starting point, here a few of them combine to form a rich representation of an apple.

As mentioned the utility of this arrangement seems to be primarily redundancy, a deaf individual can still form a successful apple representation because there’s more to an apple than the light it reflects.

The distributed nature does seem to go further than basic sensory modalities, there is rich communication between them ( connections between representations ) as well as more nodes in the form of abstract hierarchical representations ( the idea of an ideal apple ) and even more connections ( Apple pies and Apple computers ).

Connections galore! (in black) besides the already mentioned sensory map representations, visual representations can connect to the other modalities like gustatory ones and also to other acquired nodes like fruit, food or related things which in turn have their own connections and relationships. Thousands if not millions of connections tying representations seem to exist in your brain, the detail of how they are formed is the subject of memory acquisition and consolidation. 

This nested and distributed effect is easy to experience. If you can’t remember a name, fact or other bit of information ( yet you know you know it ) it is usually helpful to start from what you remember about it, the context might be intact and you can later retrieve the missing piece.


We recognize numbers, letters and speech even if they are presented to us as if they had been dragged through the mud: missing pieces, blurred parts, warbled or thick pronunciations, yet remarkably we make sense of them without much effort. To code for this feature we use the sparse and distributed elements along with consolidation over time and new experiences. For instance, do you remember the symbol we just used as an example ? ( if not take another glance )

You should have no problem recognizing the same symbol yet you have never seen these specific ones, same goes for hearing the word apple in a different voice, slurred or with a heavy accent.

Another enigma we have yet to fully crack, but current AIs have made a niche in storing and retrieving invariant and useful things like text and speech.


This was just a simple overview of the building blocks of the neural code, left out are many other things. For instance all of this happens through time, be it an instant or a lifespan, it also happens in parallel and the way we perceive through sensory cells ( the computations) represents a new level of complexity, if we were to go down yet another level to the fascinating interactions and behavior of groups of neurons it can be too much, at least for an overview, but what about emulating them ?

The good news is that we are well on our way to emulating these principles as such, in computer science and commercial applications hierarchical, sparse and distributed representations are routinely used for all kinds of tasks. The Invariant task has recently been employed along with Neural Networks and Machine Learning in better object and pattern recognition and more mundane things like search but we’ve yet to achieve the level of detail and complexity human brains posses. There is also the issue of higher cognitive functions which we have barely started understanding and emulating.

Following these complexities it seems reasonable to understand that raw processing power might not be enough, the complex and mostly unknown code that is happening on biological or artificial hardware is also needed.

Where to now ?

For many the end goal is to emulate and better understand human intelligence & cognition and then translate those findings into mostly commercial AIs. Artificial Neural Networks (ANNs) for instance share a background with biological concepts but soon diverged into its own speciality and rich language, newer types of AIs might emerge from new understandings in neural code.

For now we need to work on the fundamentals and to this effect we need to understand how the brain acquires, stores and gives meaning to our world in a systems view; I hope these few paragraphs help you understand some of the building blocks and challenges.

Thanks for reading.

Further reading :

I think a good overview of neuroscience is helpful both for inspiration and to better understand more advanced concepts, here’s 2 such places you can start:

Kandel, Eric R., and Sarah Mack. Principles of Neural Science. McGraw-Hill Medical, 2014.

Purves, Dale, et al. Neuroscience. Sinauer Associates, 2018.

The systems view of the brain is a work in progress, as such knowledge is dispersed throughout, here are 2 good starting points:

Rhythms of the Brain ( Buzsaki )

Networks of the brain, Discovering the Human Connectome ( Sporns )

Memory is somehow better understood, but critical pieces are still missing:

Gluck, Mark A., et al. Learning and Memory: from Brain to Behavior. Worth Publishers, 2008.

On the other side of the table AI is a fast changing field especially when it comes to Neural Networks, but that’s not the whole story, the field as a whole consists of a myriad algorithms and techniques for intelligent behavior, a good introduction:

Artificial Intelligence: A Modern Approach. S.n., 2010.

For up to date Machine Learning and ANN tools and techniques the web is probably a better resource these days.

The Startup

Get smarter at building your thing. Join The Startup’s +724K followers.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store