Illusions & Reality
How understanding illusions helps us design a better reality
Why do we love illusions? We see something we can’t quite explain, and it makes us smile. What just happened? How did they do that? Like most of you, when I see a good illusion I ponder these questions. But I also ponder other questions: What makes an illusion an illusion? And what can illusions tell us about how to design better experiences, to make a better reality?
Something unexpected this way comes.
When you watch an illusion, something unexpected happens. I like the unexpected. A good illusion and a good joke have the same fundamental construction — you experience a familiar sequence of events, and then . . . surprise, there’s a twist! Not only is what happens not what you expected, sometimes it’s not even something you believe is possible.
We know how the world around us works, and we’ve internalized the “rules of the world” deeply in our minds — gravity is always pulling down, two things can’t occupy the same space at the same time, objects in movement have momentum, objects cannot become invisible, a cut object cannot become uncut, and thousands of others. When we see an illusion, something isn’t following the rules.
It’s worth pausing and reflecting on what we do when we’re not observing an illusion. Most every moment of every day we use our knowledge of these rules of the world (also known as Physics) to understand what’s happening around us, and predict what’s going to happen next. The swinging bat will hit the ball, the snapped fingers will make a sound, the unplugged lamp will go dark. We are all constantly sizing up the situation and predicting what will happen next. A good illusion violates the premise developed in your mind of what’s happening. It’s not following the “reality” you’ve just fabricated to make sense of your experience.
It’s your reality
Did I just say you fabricate reality? Yes, I did. In fact we all do, all the time. Without getting too philosophical or existential, I believe the creation of reality is the essence of consciousness. Don’t get me wrong; I’m a person of science and I truly believe the tree falling in the forest does make a sound, even if we’re not there to hear it. But the hearing of that sound, and the seeing of the tree fall, and the feeling of the shudder of the earth, and maybe even the smelling of the freshly shredded wood — when we experience those things coherently, our mind tells us “this is real.” That’s when we create reality in our minds. And if you’ve ever had an argument with another human being, you know how tenaciously we hold on to our own reality.
Making sense of it all — with all of the senses
Coherency is about how the information from all our senses fits together with our reality. It also has a lot to do with timing. Have you ever watched a movie with the soundtrack out of sync? You may not even be able to tell if it’s ahead or behind the video, but if it is out of sync by as little as a tenth of a second, your mind knows that it’s just not right. In another situation, if you see someone drop a ball to the floor, and instead of bouncing it stops dead with a thud, you are surprised and know something’s not right. Of course it doesn’t take long for you to realize it’s not the rubber ball you thought it was, but a “no-bounce” ball made of special damping material.
Your brain on reality
We’re learning about our brains at a tremendous rate these days, and it turns out we know the areas of the brain that integrate these information streams and help us create our reality. The temporo-parietal junction is a primary area where signals from many sensory systems are processed, and it integrates information from both the outside in the environment and from inside the body.
Here’s a simple exercise: Touch your two index fingers together. Your muscles take action, your proprioceptive system tells you where your fingers are in space, your eyes confirm their position, and when your fingers make contact the nerves in your fingers sense the touch. You might have even heard a gentle tap. Yes, your mind says. Everything agrees, and all is right with the world. Think about how multi-sensory that experience was.
Making the impossible happen
But what if you are in a situation where everything you are experiencing — step by step, sense by sense — fits together just as you expected . . . and then you experience something completely unexpected. Not just unexpected, but impossible. We call that magic. Magic is the term we use to describe illusions when we want to sound a bit more mystical.
We hear a lot about optical illusions, but I’m sure you realize by now that illusions can be based on any and all of our senses. There are also illusions based on sound, on taste and smell, and on touch. In fact, most illusions are multi-sensory. This National Geographic clip uses the motion bounce illusion to show how sound changes what we see.
This particular type of illusion, based on auditory-visual interactions, is called the McGurk effect, and the best examples show how sight changes what we see. We regularly use sight to disambiguate what we hear when another person speaks, as in this example about the McGurk Effect. As it turns out, we are all lip readers.
Getting touch in reality
Touch is a particularly complex and often unappreciated sense. Unlike vision and hearing which are essentially single channels, touch is comprised of multiple systems, with different nerves for surface stimulation (such as vibration), pressure, temperature, body position, and more. The science of touch is called haptics, and Tanvas is a startup which is creating new haptic interactions.
For example, Tanvas is exploring ways in which the properties of a flat surface — specifically the coefficient of friction between your finger and that flat surface — can be dynamically changed, creating the illusion that it has textures or features.
By itself, this creates an eerie experience, because you know that it’s flat. But if this flat surface is an LCD screen, and you combine the changing surface friction with matching images (such as an animation of a switch or a visual texture), something special happens. What you see with your eyes and what you feel with your finger may have completely different sources, but they combine in your mind. The image is real, and the friction is real, but the illusion is created that they are coming from a single source — that the image is a tangible, physical object with which you are interacting. Done well, it quickly goes from feeling like an illusion to just feeling right.
It feels like reality. A better reality.
Making magic real
Arthur C. Clarke, the brilliant futurist and science fiction writer (including 2001: A Space Odyssey) once said, “Any sufficiently advanced technology is indistinguishable from magic.” With our growing knowledge of human perception, and our new technologies to stimulate the senses, it’s an exciting time to be making new illusions — and we now know that the sense of touch is an integral part of this story.
We are starting a conversation about this magical intersection between emerging technologies and the human senses, where we have the opportunity to make better interactions with the increasingly tech-enabled world around us. If we keep the human experience at the center of our work, we can make better illusions and ultimately, better realities.
