Why “machines cannot understand” is a wrong way to think about the potential of AI — on substrate independence

Karl Koch
Karl Koch
Sep 3, 2018 · 11 min read

Note: Again, more of a write-up of thoughts rather than an academic paper. Hope you enjoy it and always looking for critical and constructive feedback. Motivation for post at bottom of entry.

Exec Summary: When people claim that machines can’t understand, they don’t underestimate machines, they overestimate humans. “Understanding”, as used by us, is only the degree of confidence humans feel in their mental models — and that’s usually highly overconfident. Humans don’t actually understand (cognitive) most things they feel they understand (affective). As the process of building mental models seems to be achievable on any substrate, the only difference lies in the capability to feel emotions. Claiming that machines will “never be able to understand” is therefore solely a statement on a machine’s capability to have emotions. And that capability, in my opinion, is also substrate independent. For context, I write about this because it seems important not to fall victim to wrong signals on AI progress that are based on being “human like” along dimensions that do not correlate with AI capability. This could lead to frequent disappointment in AI progress and consequentially to underestimating powerful AI once it actually arrives, which could be very dangerous.

Disclaimer: I am not clear on timelines for AGI arrival and would refer to the known studies out there. I am also not dealing with qualia/ suffering potential of digital minds. This is highly relevant, but as we know so little about sources of consciousness I am working with the assumption that intelligent machines are thinkable without consciousness/ experienced emotion.

Sources: The original idea of substrate independence is, unsurprisingly, as old as AI research itself, however at this point a shout-out to LessWrong, where I read most about the topic. The idea of “understanding as data compression” originates from Gregory Chaitin’s paper on “The Limits of Reason”. The rest is a mixture of the usual suspects like Yuval Harari, Max Tegmark, probably Brian Tomasik, and other LW posts.

Let’s do this!

“Machines can’t understand” — background to the question

Before going into the actual questions, let’s quickly talk about the base of this article: We can think about animals, including humans, as agents that try to optimize for some goal. That goal, in nature, is maximizing the replication of their genes (how could it be otherwise in an unregulated system?). To do so, we need to navigate our environment in such a way that we reach our goals. We can think of nervous systems as the actuators for that process: They are basically algorithms that, at a base level, resemble “if-then” models. For very basic animals like invertebrates, that means “feel pain (nociception), move away”.
There is a strong evolutionary payoff from navigating the world more effectively than adverse forces in the environment, which made nervous systems more complex and performant over time. As they got more complex, the brain developed — shaped by its environmental constraints like e.g. mortality of the host, energy availability, or allowance for longer periods of being hardly able to do anything (childhood). This set of constraints left us with an amazing machine that can create very sophisticated algorithms to model the world around us and act upon them. It seems important to note here that “algorithms” does not imply that these are “logical thinking” type processes, consciously experienced and/ or separate from emotions. We can see emotions as parts of this algorithm that gives us indications about optimal behaviour — some acquired vs. some deeply engrained as intuitions (“boss comes to my desk with a paper box that probably means I’m fired ->fear” vs. “snake -> fear”).
Restraints like energy availability also lead to the need for less than gigantic working memory and the consequent use of heuristics (which can be biases) to quickly and energy-efficiently react to the environment.

“Machines can’t understand” — what do people mean by this?

As we just saw, we can think of the brain as an information processing system of some sort. So what do people mean when they say “machines can’t understand/ will never be able to understand like humans do”?

People seem to mean several things:

· Direct: There are types of information processing in an organic brain that cannot happen in a silicon-based one.

· Implied: These processes are necessary for creativity and superior intelligence across a variety of domains

So what are these types of processes that constitute understanding? Think about the last time you understood something. What happened in your head? What did it mean for your actions? What did it feel like? To get us all on the same page, remember the following 14 numbers and retell them a minute afterwards. You have 10 seconds.

1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192.

I’m guessing you had an experience along the lines of “damn these are quite a few bits (35 to be exact) to remember in such a short time”. As random numbers are notoriously difficult for us to remember, this maybe got you a little unnerved. Then you noticed, with a small sense of pleasure, that all numbers were integers, doubled from 1, 14 times. Easy, understood. If I now asked you to recite them, you could, given your pre-existing knowledge of multiplication, solve this by using a much shorter operator in your head. The only numbers you need to remember are 1 (start), 2 (times previous), and 14 (total). [I’m not a computer scientist so no idea about whether Kolmogorov complexity would already be lower at this number of digits, but at some point it should be, which is the point]

What is understanding really?

Cognition: What happens to the information we “understand”?

In this example, you compressed information you were presented with in such a way to still make sure you could derive the original information reliably. You saw the information, reduced it by building a model of the information based on key data points (i.e. building relationships to other concepts/ models), scaled it up to test it and, upon successful regeneration, stored the model. As such, all understanding is relational in nature. This is the cognitive part of understanding and different from pure knowledge: Knowledge is not compressed in any way, it’s stored in its original size. Referring to our intro, one can see how such a compression of information makes sense from the perspective of navigating our environment at minimum computational (=energy) cost.
This also means that what we call “deep” understanding refers to possessing mental models that allow for maximizing accurate output information at minimum stored/ input information. Importantly, this does not mean having a model that can explain everything but to be accurate in predictions of what is true*. As an unscientific example, “the nature of suffering” in Buddhism is often referred to as a deep concept, as truly understanding it, according to anecdotal evidence, leads to immediate enlightenment. As another example: Understanding basic concepts of military strategy, e.g. Sun Tzu’s rules on “know thy enemy and thyself” can, if truly understood, save one 1,000s of hours learning different battle situations off by heart.
If you’re neither a Buddhist nor a Master of War (a small subset of my presumable audience), a nice, scalable example could be navigating your city: Imagine there are 100 streets in your city, crossing each other at random. If you only know two streets and their crossings, the amount of paths you can generate from any point A to point B in your head is very small — you would have to store all other paths as information like “right-left-right-left…” to get from any one point to any other one. At 100 streets crossing each other in some way, you can imagine the number of paths you would have to remember to cover all possible routes from any one point to any other one. If you have a model of these streets in your head (a deeper understanding of your city), you can generate true paths with some computational cost — which is however presumably much lower than storing all of the information!

Information, maps, crativity, and (hand-model)

Creativity lies herein, too. Accurate mental models can, from one deeply compressed state, extrapolate many other, new bits of (ideally) accurate information and discover relationships to other models, compressing them again along their respective relationships. This can also come from auto-generated information (think of the philosopher sitting in his chamber without any input outside of his own thoughts), that is however necessarily again compressed in mental models. Probably uncontroversially, I’d propose the generation of information and the connection of generated information to other models is creativity.

During and after this compression of information (that is model creation), we assign (implicit) confidence scores to our models, i.e. both how confident we are that the relationships in our model are true AND to what extent we fully understood the true relationships. To give an example: We might fully understand the argument a friend is making but vehemently disagree with it. At the same time, we might strongly believe relativity theory is true but not have an understanding of its nitty gritty workings.

As you probably noticed, we did not mention the substrate a single time here. What we’re talking about here is pure information processing. Which part of this do you think requires a biological data compression machine (for superior performance)? If you have an idea, please let me know — because I can’t see one.

What is key here: The cognitive part of understanding entails identifying key data points in input information that allow for creating meaningful connections and models to recreate the relevant parts of the input information (and possibly additional info, the process of which we call creativity) accurately. This process does not seem to require a biological substrate.

Affect: What happens to us when we understand?

“Understanding”, how we use it, is not only the abstract compression of data. It feels like something to understand. What we feel when we understand something can, as I think, nicely be illuminated from the role of affect/ emotions in biological life:

For one, there is the reward function of emotion. One can witness that reward for heuristically optimal behaviour (from an evolutionary standpoint). In this case, this refers to the pleasure of understanding something — as understanding allows us to navigate the world more effectively in line with our goals, which should be encouraged!

Secondly, emotions also guide our behaviour more directly, as we can “feel” our confidence levels, e.g. when making a point in a debate or when preparing a lecture. “Have I understood this? I still feel a bit shaky/ uncomfortable on this, but very relaxed about that”. This means we will take different actions based on different confidence levels.

To some extent, emotions do actually seem to be substrate dependent — at least we know that certain chemicals make us feel a certain way (by triggering certain information processing in our brains). One could however also imagine the same effects emotions fulfil (reward & directing action of an agent based on confidence in models) without any subjective experience.

What is key here: Emotion/ affect is the reward and/ or steering function of the brain, operationalising cognition to create optimal behaviour towards reaching goals. Again, both seem like critically important functions for a self-optimizing agent — could however also be solved without emotions.

Bringing it together: Understanding is an illusion

So what? What’s the point if our emotions are good “actuators” and trainers for optimal behaviour? Well: Our emotions related to our confidence level, i.e. our sense of security, are often extremely wrong. Just look at Kahnemann’s and Tversky’s work or think for yourself: “What is the simplest thing I could explain to a person from the stone-age how to build? Not broadly, in detail”. I’m betting you you’ll be shocked how many things you feel safe about “understanding” are mostly false sense of security — your “understanding skills” are not actually as good as you think they are, you just feel safe about them.

To give another angle, you may still say “hey — there’s a difference between finding correlations in information and storing their relationship vs. actually detecting causality”, which is also often implied in “understanding”. This is exactly the inaccurate emotional processing of confidence I mean: Based on the problem of induction, it is not possible to nail down causality with 100% certainty. Humans simply equate 99.999…% certainty over sequences of events with a perceived 100% certainty and call that “detected causality”. That of course makes total sense for us to navigate our world as they are quasi-equal, but it is a difference in emotionality, not a qualitative difference in understanding. You could navigate (by understanding) the world equally well without assigning a (wrong) emotional value to your confidence levels.

So this is it: “Understanding” as used by us is only the degree of confidence felt in mental models — and that’s usually highly overconfident. Humans don’t actually understand (cognitive) most things they feel they understand (affective). As the process of understanding seems to be achievable on any substrate, the only difference would lie in the capability to feel emotions. Claiming that machines will “never be able to understand” is therefore solely a statement on a machine’s capability to have emotions. If they can do so (I believe they can) is another question, but even if they could not, it would not limit their level of cognitive understanding.

*This does not mean that a Theory of Everything, aka a single model that allows to prove anything true from a finite set of axioms, is necessarily true. It might well be that these axioms need to be infinite, which would correspond to new information being added to the system as new axioms accordingly. I am not a mathematician so this is only my layman understanding of Gregory Chaitin’s work, who basically derived the above from Turing’s halting problem.

Motivation for this post:

When questioned, quite a few people (my guesstimate >75% of people not working in an AI-related field from my experience) believe that there is a qualitative difference between organic and silicon-based computation, understanding, creativity, and ability to reach goals (call this intelligence) — usually with the connotation that the organic one is superior on a (goal-post shifting) set of increasingly complex problems. This belief in difference is called substrate-dependence, referring to different “platforms” for computation, in this case organic vs. silicon-based substrate.
Such beliefs would be expressed along the lines of „well, machines will still never really understand what they are doing“, or “show me a robot that can feel hate and I’ll get scared” or “prove to me a robot can sense beauty/ love and I’ll fear for my job as a musician/ care-taker”.

This seems dangerous as it can lead to a false sense of security and poor general public understanding of AI progress: Misunderstanding the (presumable) roots of intelligence could plausibly lead to wrong perception of relevant signals for AI capability progress. That means relevant signals on AI progress could be underappreciated. For example, last year’s AlphaZero evidence on generalness probably leading to greater performance also over narrow domains would be such a strong signal that was pretty unappreciated** (although I also definitely wouldn’t claim I could appreciate even a small share of relevant signals [yet — staying positive])
At the same time, signals indicating similarity of AI to human cognition by structure or statement* (like passing limited Turing Tests through fine tuning, indicating human emotions/ using rewards in similar structures as emotions, or having “cute eyes” and being “friendly”) could then be easily overvalued and, given sufficient frequency of these non-signals, lead to a boy-shouts-wolf situations. Such frequent disappointments of the public regarding AI progress are probably detrimental for a safe and beneficial development of AI. As an example for such a misguided signal interpretation, Sophia by Hanson Robotics, which received citizenship in Middle-Eastern Country, sparked a craze over apparently being highly intelligent as she solely qualified for her citizenship for her human like behaviour. Its actual performance is far from human-level.

I therefore tried to explain why people should believe in substrate independence along the example of “understanding”, which people still seem to get confused about.

*having an AI compute similar to humans may be a good signal for progress, as humans are comparatively intelligent (an AI might already be at speed superintelligence levels then). This would however only be instrumentally so and not as “the only route” towards intelligence. Also, this is obviously not referring to similarity in performance, which would be the goal already.

** See Eliezer Yudkowsky’s comment on that here, worth reading the whole thing: https://www.facebook.com/yudkowsky/posts/10155992246384228?__tn__=K-R

Karl Koch

Written by

Karl Koch

Interested in AI (safety), broader tech & science, consciousness, ethics, and strategy

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade