Beyond Vision: Sound Design as Sensory Design

Orchestrating healthier and more beautiful digital experiences

We’re distracted, depressed, and overwhelmed. The digital experiences that were once fun, delightful, and helpful now feel like a burden, an always-on state that we hope to escape.

—Albert Shum, Microsoft CVP of Design

Our sensory experience of the digital world is out of whack, and it’s making us less happy and less healthy. We know that digital technology can be a source of cultural disruption, but it’s also a source of sensory disruption — we feel it in our bodies.

The digital age is first and foremost a visual age, experienced through the eye. Our screens deliver an unrelenting stream of information — notifications, text, graphics, video — that feeds us quick hits of visual energy while ignoring our other senses. Working alone, the eye is limited to the surface of things. The digital world has a lot to see, but without deeper sensory engagement, there is much less to feel (both physically and emotionally).

The world is not a flat screen; we are designed for a richer existence. Our brains and bodies are hungry for different kinds of information — and for layered multisensory experiences — yet we address them in the digital world almost exclusively through the single visual channel. This sensory imbalance contributes to emotional imbalance as we feel more distracted, depressed, and overwhelmed.

We can begin to heal this disparity by embracing a more intentional and holistic approach to designing for the senses. Sound is a powerful and flexible sensory element at the heart of a more holistic design strategy.

Sound is feeling. Beyond its visceral emotional texture, sound plays an important role in organizing key constellations of sensory experience that add depth and dimension to the digital world:

• Our perception of tactility and texture
• Our physical and emotional response to language
• Our perception of the passage of time
• Our sense of place and environment

A holistic approach to sound can help us create digital experiences that are more functional, inclusive, beautiful, and soulful. To accomplish this, we must first shift the way we think about sound design.

A new kind of sound design

When sound does occur in our digital world, it’s too often an afterthought. Poorly designed sound is an invisible stressor that creates anxiety and disrupts the flow of attention. We now routinely expect beauty and excellence in visual and hardware design, but we’ve been conditioned to accept the idea that our devices just naturally sound kind of annoying.

One reason for the current sensory imbalance is that we haven’t had a good model for conceptualizing the role of sound in modern digital experiences. For two decades, the primary way of thinking about sound design has centered around alarms and alerts. An alarm-based model maintains that to be effective, UI sounds need to be heard, and to be heard, they must capture our attention. This model has roots in the early days of mobile technology, where it made more sense given the way devices were used and their limited audio capability.

But as the role of digital technology in our lives has become more sophisticated, some of this old-school way of thinking about sound design has become the “common sense” of our discipline. Assumptions of an alarm-based model are so ingrained that they are often not recognized or questioned, even by experienced sound designers.

While sounds must be heard or felt at some level to be effective, this focus on attention grabbing has led to a modern soundscape filled with apps and devices producing annoying sounds, competing to be noticed against an ever-noisier background. This is an unpleasant, fragmented, and semi-functional audio world. Noise pollution, a global problem that negatively affects mental and physical health, has expanded to the soundscape of technology.

Time and attention are among our most valuable resources. The last thing we need is more sounds competing for our attention.

To support the evolution of modern digital experiences, we need a sound design practice based on a deeper understanding of sound in human perception, and on the many ways that sound conveys information and shapes experience. We need a new paradigm based on the way people actually feel sound-in-context.

Designing across senses

Contrary to conventional thinking, our senses are not separate channels; they continually converge and overlap. We always experience sound as part of a multisensory web of perception. Because we respond to sound more quickly than to our other senses, it plays a crucial role in orchestrating the flow — and the physical and emotional texture — of sensory experience. Sound shapes what we see, what we touch, and how we feel about it.

Understanding sound as sensory experience means tapping into the wisdom of these intuitive processes to design experiences that resonate more deeply with the body and mind. We approach sound as a way of intuitively knowing, as an embedded element in the web of sensory experience that helps us make sense of the world. We call this the sensory sound model.

The sensory sound model is less about sound and hearing, more about sound and feeling.

It’s less about individual, discrete sounds that demand our attention and more about sound that we perceive as seamlessly integrated with experience. It’s less about a series of single, isolated sonic events and more about the connected soundscape we are collectively creating.

In sensory sound design, quiet is beautiful. We design silence. In fact, quieter sounds are often the most effective. As louder sounds fight for attention, they create distraction and resistance. But quiet sounds blend more seamlessly to help choreograph a calm, natural, and fluid rhythm of experience. Instead of competing to be at the center of attention, sensory sound works gently from the periphery, expanding the depth of cognitive and digital space.

Feeling sound

Our design approach has its roots in the physical and psychological properties of sound. As physical energy, sound is vibration that travels through the air, part of a spectrum of vibration that we feel and hear — not just with our ears, but with our bodies, bones, and skin. As psychological energy, sound is a carrier of emotion and meaning, and interacts with memory.

Sound is physical energy (vibration and sensation).

The vibrations of sound penetrate the body and connect our outer (physical) and inner (emotional) worlds. Sound connects our physical design language and our emotional design language.

Sound is psychological energy (emotion and meaning).

This can seem abstract, but our experience of sound is primal and intuitive. We’re not aware of it because we don’t need to be. Our brains have evolved to process sound quickly and unconsciously. The sounds around us are always shaping how we feel, how we process information, and how we interact with our environment and one another.

In a digital world dominated by vision, we’re always in our head. Understanding sound as physical and emotional energy helps us create digital experiences that are more grounded in the body — and the heart.

Sound as touch

Sound has a haptic quality, it communicates through direct physical contact. Haptics and sound exist on a spectrum of touch. While haptic feeling involves the vibration and touch of objects in direct contact with our body, the vibration of sound is touch from a distance.

Sound and touch are both immediate, visceral, and emotional. Both are full-bodied senses that connect the outer world with our inner world.

If we think about designing sound as designing touch, we make different choices. A gentle touch can be more effective than a forceful one. The right touch at the right time can unite the digital, the physical, and the emotional.

Designing sensory sound

Understanding sound as a sensory experience can transform the way we listen, the way we think and feel, and the way we design. It can help expand our design palettes to create healthier and more cognitively sustainable digital experiences. And it doesn’t have to be complicated — it’s more about shifting the way we think about sound.

Like the early days of other media, we need to explore, experiment, and invent new combinations of sensory experience and new ways of connecting the digital world with our physical and emotional worlds. We need to develop a creative process and methodology to help us design across senses and across devices.

What you can do right now

Design isn’t abstract or outside of us, it’s something we feel. If you’re a designer, try doing a quick sensory audit of an experience you’re creating. Or think of an everyday digital interaction like checking email on your phone. Tune in to how the interaction feels for the whole body and mind, and ask yourself:

• What, if anything, do you hear? Listen especially for the quieter sounds.
• How does the sound or silence impact your attention? Your feelings? Does the interaction feel calm or anxious?
• What is the physical rhythm? How does it feel in the fingers, hands, and arms?
• Would someone with limited vision be able to perform this interaction? If not, how might it be designed for better accessibility?

In this post, we’ve laid out the conceptual framework for approaching sound design as sensory design. Next time, we’ll begin exploring process and methodology, with specific strategies and real-world techniques. Let us know your thoughts in the comments below.

To stay in-the-know with Microsoft Design, follow us on Twitter and Facebook, or join our Windows Insider program. And if you are interested in joining our team, head over to aka.ms/DesignCareers.