Deep Learning, Semiotics and Why Not Symbols
Any study of cognition, whether that be biological or artificial, requires a rudimentary understanding of semiotics. This article will focus on semiotics, the study of signs their use and interpretation. I shall discuss here icons, indexes, and symbols as defined in Semiotics. I will relate this to related subjects like self, affordances and mental models. In so doing, one will get a sense of the richness of information and thus accomplish a more informed understanding of the kind of representations employed in Deep Learning.
I will, however, perform a twist in the conventional treatment of semiotics in order to better explain intuitive cognition.
Semiotics defines three kinds of signs. These are iconic references, indexical representation, and symbolic representations.
- Index: experienced representation that associates a sign with the cause of the sign (i.e. the effect of the subject).
- Icon: unconscious representation of similarity between sign and subject.
- Symbol: learned representation of associates a sign independent of any similarity or causal relationship.
Those familiar with Semiotics will realize that I’ve changed the ordering here. It turns out this ordering is important as we shall see.
Indexes reflect causal relationships between the signifier and signified. These are the cues that reveal useful information to an organism. Icons reflect similarity between signifier and signified. The twist here is that it is not icons which are the most primitive sign but rather it is the Index. Symbols reflect abstract references that are independent of similarity or causal relationship. Indexes give rise to icons which give rise to symbols. This is different from the classical ordering of semiotics.
Let’s subscribe to this notion that embodied learning is essential to general intelligence. That is, human complete intelligence requires interaction with environments to learn. So far in our discussion of signs, we have confined this to external things. Let’s take it up another level and consider signs that are internal mental models that reside in minds. From this perspective, we can gain an understanding of the progression of higher intelligence capabilities.
It indeed is interesting that this classification of signs maps roughly to how the human biological brain has evolved. The instinctive reptilian brain recognizes indexes that are recognized as cues to behavior. The wings of butterflies and moths have evolved to appear like a pair of eyes. These are meant to deceive predators based on a cue. The limbic mammalian brain is able to recognize similarity and thus form its own causality relationships through repeated observations. The neo-cortex human brain is able to process symbols requiring higher abstractions. A bit of a caveat, the last statement shouldn’t imply that the purpose of the neocortex is symbolic processing. The purpose of the neocortex appears to be imagination and reflection.
The most primitive level of information processing is sensory. We interact with the world through sensory patterns of sight, sound, smell, taste, and touch. These sensations give us our instinctive understanding of the attributes of objects in the world. These sensory patterns are mental indexes that cue an organism to further action. Everything we understand about objects (more specifically spacetime, this is proximity, directionality, composition, and similarity are expressible by mental indexes. In short, without mental indexes, one cannot understand spacetime and thus anything about reality. Mental indexes enable minds to predict reality by being one step ahead. What this means is that the mind only needs a cue and not the entire recognition of an object to be able to decide on an action.
Icons arise when we can make generalizations about the objects that we have interacted with and have realized that they can be treated similarly (i.e. polymorphic ). Through experimentation, we learn to recognize new causal relationships between different icons. This interventional learning leads us to develop a causal model of the world. A causal model is abstracted in the identification of icons. In identifying new icons, we recognize new kinds of cues and thus expand our own repertoire of skills. human reasoning is based on icons and their affordances.
As an example, we learn to master playing a musical instrument through thousands of hours of experience. This is the ‘amortized inference’ that delivers fast unconscious thought. These intuitive patterns are learned from mental indexes. These are the affordances that are cues the reveal the usefulness of a sign.
At the next level, we interact with the world through abstract symbols. These symbols allow us to communicate our thoughts with others as well as allow us to gain new knowledge independent of actual experience. Abstract symbols are acquired primarily through our social interactions and the use of human language. They are defined by convention and by their use. Terrence Deacon describes this evolution of our symbolic processing capabilities in the Symbolic Species where he argues that language is a consequence of civilization. Even more controversial, Julian Jaynes in 1976 in “The Origin of Consciousness in the Breakdown of the Bicameral Mind argued that language is a necessary component of subjective consciousness. Jaynes argues that our present-day self-aware minds are a recent innovation (i.e. 3,000 years old). Language is perhaps a recent invention and this implies that symbolic processing may not be an innate biological capability, but rather a learned capability.
Language is essentially the digitalization of thought, like DNA (base-4). Verbal language originates from spoken language that is composed of phonemes. A human listener has great difficulty replicating words that originate from a language that uses different phonemes. Digitalization enables error-correction in communication. Thus one unequivocal value of symbols is that it is a digitalization of thought and this enables error correcting knowledge transfer.
The other important feature of symbols is that symbols serve as names. The Pi-calculus is a compact process calculus that at its core is the concept of channel names. The Pi-calculus, like lambda-calculus, is also universal. Pi-calculus is a minimal descriptive language that describes parallel processes. Nature, of course, is nothing but parallel processes. Robin Milner the inventor of Pi-calculus in “Turing, Computing, and Communication” writes:
So we must find an elementary model which does for interaction what Turing’s logical machines do for computation. … there is a logic of informatic action, and in my view it is based upon two fundamental elements:
Synchronised action and Channel, or vocative name
I ask you to think of the term “information” actively, as the activity of informing. An atomic message, then, is not a passive datum but an action which synchronises two agents.
In other words, symbols are information about coordination. Furthermore, symbols are information about the reference. References, are the building blocks of communication and originate without the vocalization. Previously, I’ve discusses how references can originate through gestural sign language.
Human words tend to be ambiguous and their interpretation will always be subjective. I can listen to an expert and my understanding of his content would vary from someone who has more or less familiarity with the subject. The skills that allow humans to process ambiguity are the same skills that allow humans to process indexes and icons. (Reminds you that it is worth listening again lectures over a lifetime. In fact, re-reading what you previously wrote at a later time clarifies one’s own understanding. I’ll write more about how environments substitute for cognition)
The symbols that we find in logic and mathematics do not share the same ambiguity in words as found in human language. Their grammars are much stricter and do not need an entire field (i.e. linguistics) to devote to their study. The cognitive dissonance of linguistics is that it attempts to develop formal rules for language when in reality there a none to be found. The rules of language have evolved through use and conventions. It is through our flexible and adaptive intuition that we are able to compensate for the inconsistencies and ambiguity of language. Formalization is a straight-jacket when none is actually needed. The human mind is motivated to find structure in complexity even if there is no structure to be found.
Sabine Hossenfelder wrote the book “Lost in Math: How Beauty Leads Physics Astray” which explores the same human bias in the field of physics.
Symbols require interpretation to lead to understanding and this interpretation requires a connection back into an index or an icon to be comprehensible. A symbolic processor is capable of discovering newer viewpoints through term-rewriting, but it doesn’t arrive at an understanding in the absence of symbol grounding. Symbolic processing is extremely useful in discovering new knowledge because it reveals a relationship that previously was not explicit.
Symbolic systems like logic and mathematics have their own complexities and are extremely valuable thinking tools for understanding our world. I would argue, that despite their failure in analyzing complex non-linear systems, those formal methods remain valuable for advanced kinds of systematic reasoning. One can argue that AlphaGo with its coordination of rational and intuitive cognition is one example of advanced systematic reasoning. Formal symbolic processing has its place in intelligent systems but is not required for general intelligence. General intelligence is sloppy and messy and symbol processing is, therefore, the wrong tool for the job.
To have any understanding of the world, intelligence should first understand indexes, work itself up to icons and reinventing new indexes then finally to abstract symbols. The failure of GOFAI is that it assumes the existence of symbols from the beginning and thus unable to bridge the connection back to indexes and icons. This is the symbol grounding problem that has continued to elude symbolic processing approaches.
Stevan Harnad explains this elusive problem in “The Symbol Grounding Problem” where he writes:
there is really only one viable route from sense to symbols: from the ground up.
Good Old Fashioned AI (GOFAI) fails because it is a top-down approach and it is likely to be exceedingly difficult to align the meanings of symbols coming from the top and semantics coming from the bottom. Cognitive development in humans happens by building up icons and indexes from the ground up. How these internal models align with conventional symbols requires learning. How does a person know how to identify the color red? The symbol ‘red’ must first be learned to be linked to the perception of the color red. Kevin O’Regan explores this idea in grander detail in “Why Red doesn’t sound like a Bell?”
Let’s now take this mental model of signs to an even higher level. That is, let’s consider signs with its alignment with that of the subjective self. How do signs serve as a representation of self?
Anil Seth describes five kinds of self, the bodily self, the perspectival self, the volitional self, narrative self, and the social self. Selves are the internal mental signs that we associate with the different kinds of self:
There is the bodily self, which is the experience of being a body and of having a particular body. There is the perspectival self, which is the experience of perceiving the world from a particular first-person point of view. The volitional self involves experiences of intention and of agency — of urges to do this or that, and of being the causes of things that happen. At higher levels, we encounter narrative and social selves.
We cannot fail to recognize that the bodily and perspectival selves involve sensory patterns as represented by instinctive indexes and icons. The volitional selves and its direct manipulation and interaction with objects in the world lead to the experiential learning of new mental indexes (causal models). Our entire feel of consciousness, according to the O’Regan is based on our narrative selves. Our language is based on our social selves that interact with society to learn the conventions of human interaction and language.
To begin, the reflection of oneself through perhaps a mirror is an iconic symbol of one’s self. Only animals of higher intelligence are able to recognize their selves in their reflections. This reflection is a visual perception of one’s own bodily self. When we perceive the effect of our selves in our environment. That is when we move an object within our presence, we are aware that we were the direct cause of that movement. This is an indexical reference to yourself.
We also perceive other selves and become aware of their continued connection with other objects despite their absence. That is, when someone leaves their footprint in the sand, we are aware that this is a consequence of that person and are reminded of that person despite their absence. This is an indexical reference of another self.
When we are given a name, we learn to treat this as a symbolic reference to yourself. We eventually learn that other selves have their own associated symbols (i.e. names). This is the most rudimentary notion of a symbol representing self.
Eventually, we begin to develop our own narrative of our own selves. These narratives also become an introspective model of our selves. This gives rise to the illusion that we’ve identified as being consciousness. We employ these narratives of self to communicate with others who we are.
As we interact with other selves and with society, we begin to develop our social selves. Our social selves are describing our ongoing relationship with others and society. The invention of human language originates from societal interaction. I discussed previously how the dexterity of the human hand is one of the key capabilities that lead to human intelligence. The key principle here is that thinking originates from interaction and thus complex thinking originates from complex interaction.
We have thus identified three levels of signs. The physical signs, the mental signs and the signs of self (i.e. subjective signs). At each level, there is a notion of the indexical, iconic and symbolic. These signs may or may not be expressible in our language as words. The relationship of these signs with each other is captured by analogies. The collections of these signs and their relationship represent our models of the world.
Which leads us to the question, why do we need models? Joshua Epstein has an essay as to why we model:
- Explain (very distinct from predict)
- Guide data collection
- Illuminate core dynamics
- Suggest dynamical analogies
- Discover new questions
- Promote a scientific habit of mind
- Bound (bracket) outcomes to plausible ranges
- Illuminate core uncertainties.
- Offer crisis options in near-real time
- Demonstrate tradeoffs / suggest efficiencies
- Challenge the robustness of prevailing theory through perturbations
- Expose prevailing wisdom as incompatible with available data
- Train practitioners
- Discipline the policy dialogue
- Educate the general public
- Reveal the apparently simple (complex) to be complex (simple)
In summary, we model so that we can gain access to a smorgasbord of advanced thinking tools. Humans are able to improve on their models because they can introspect the models that they create. The current crop of Deep Learning innovation (AlphaZero included) is unable to create explicit models of their domain and thus unable to perform the tasks enumerated above. Counter-factual reasoning is required to make the next leap higher.
However, despite having these advanced mental models, our own mental capabilities are extremely fallible. Philip Johnson-Laird describes in “Mental models and human reasoning”:
Reasoning is more a simulation of the world fleshed out with all our relevant knowledge than a formal manipulation of the logical skeletons of sentences. We build mental models, which represent distinct possibilities, or that unfold in time in a kinematic sequence, and we base our conclusions on them.
To summarize, our natural understanding of the world is through experience and it is of unconscious intuitive nature. Humans are intuitive thinkers. Rational and symbolic thinking is not intrinsic capabilities. Rather we create simulated mental models of the world and we work our way through these models in an intuitive way rather than a symbolic or logical way. This is why the capability model described below:
should be based solely on iconic and indexical signs and not symbolic signs as proposed by GOFAI. Symbols are useful for symbolic processors, however, if we are to achieve human complete systems, then we need to focus on representations that support analogical thinking. This would imply that the signs that are useful are those that are iconic and indexical.
Further Reading
The Proper Treatment of Symbols in a Connectionist Architecture
Semiotic Schemas: A Framework for Grounding Language in Action and Perception
https://www.youtube.com/watch?v=Dq7haK5I2HY …
https://www.youtube.com/watch?v=jDyra0RH-Xs&feature=youtu.be&t=2143 …Kurt Gödel & the Limits of Mathematics
