Exploring Multimodal Design — An Adobe XD Tutorial

Frederik Goossens
Sep 4, 2019 · 7 min read

A mother getting ready to prepare dinner at home opens a cooking app on her iPad or SmartTV. Using touch and voice she interacts with the app, telling the video to pause, fast forward, or replay a specific section. If she chooses, she can tap on the screen instead of using her voice.

Using multiple modes of interaction is what multimodal design is based on.

Before we explore multimodal design, let’s begin with a basic understanding of two types of interactions: computer-to-human and human-to-computer. Each of these interactions includes various modalities such as voice, touch, and taptic feedback.

Computer-human Modalities

Computer-human (or computer-to-human) modalities help a computer understand what the user wants. The most common computer-human modalities rely on vision, hearing, and tactile abilities. A few examples are computer graphics, audio playback, phone vibrations, and smartwatch taptic feedback.

Human-computer Modalities

We’ve invented several ways to interact with computers. A few examples are keyboards, mice, touchscreens, trackpads, and speech recognition. In this case, users rely on various mechanisms to communicate with and command computers.

More complex examples are accelerometers, gyroscopes, and magnetometers that help with motion detection. Think about playing a game of tennis on a console and using the gamepad to emulate the movement of the racket. This brings many more opportunities to create a unique and engaging [multimodal] user experience.

Nintendo Switch controllers being used for in-game archery are an example of multimodal interfaces. (Gameplayonly)

Why Multimodal Design

The idea behind multimodal design is to combine multiple modalities in order to improve a product’s user experience. As everyone uses products in different ways and in different contexts, users are provided with several feedback mechanisms and given multiple ways to interact with their computer.

Designers make life easier for users by incorporating and automating actions through different modalities. If there was only one modality mechanism, it would negatively affect the user experience and the design would “fail” in the mind of the user.

An example is a car infotainment system. Most of these systems allow users to interact with voice and touch. When driving, the obvious choice is to use our voice for making a phone call or navigation, but when parked, it’s most likely easier to use the touchscreen or a scroll wheel to interact with the system.

Here are a few more examples we commonly find in multimodal design:

The Apple Watch is a great example of multimodal interaction design. (John Sherrod)

Multimodal design is also helpful when designing for people with certain limitations and disabilities.

A Simple Multimodal Design Example: The SmartHome

Robert approaches home after a long day. His home automation system is triggered once he gets within a mile of his garage. The system recognizes that he has arrived and starts a sequence of automated actions. For example, turns on the lights, adjusts the heating and cooling, and deactivates the alarm system.

Next, Robert could either use a remote control or ask the AI-enabled assistant to turn down the heat when he walks in.

A Complex Multimodal Design Experience: Health

We are now capable of capturing more complex inputs from users using smart devices. We can measure inputs such as stress levels, heartbeat, sleep cycles, water intake, and in the near future, glucose levels.

Once these inputs are stored, devices and services such as Fitbit and First Aid by The Red Cross provide valuable, lifesaving warnings in the form of a vibrating alert, a taptic “tap” on the wrist, or an audible alarm.

This is a more complex use of multimodal design because the balance of inputs to outputs needs to be calculated correctly. The design must not give any false alarms, potentially sending the user into a panic.

Whether designing simple or complex multimodal experiences, one of the best ways to better understand multimodal design is to start designing with it in mind. Let’s look at how we can achieve this using Adobe XD.

Prototyping a Multimodal Experience in Adobe XD

Adobe XD, a popular UX design tool, recently added voice commands and playback to its arsenal of features. By taking advantage of them, we can add modalities such as speech and audio playback to create a multimodal user experience.

As an example, let’s prototype the mobile journey for a cooking app. A chef is showing how to cook a steak, and people can tell the app to pause, repeat, or continue using voice or touch.

We first prototype all the screens that are needed to illustrate the experience:

All screens for the Alexa skill have been designed and mapped out.

Next, we add a voice command that will emulate a voice modality. In prototype mode, we start by connecting the first and second screen. Then we select voice under trigger and write out an utterance under command to trigger this transition. If we want to add two or more voice commands, we will need to add a connector for each one.

Adding a voice trigger to the cooking app in Adobe XD prototype mode.

For the next screen, we want the system to answer the user. We do this by creating a time trigger and adding speech playback under action. Since we want to create an immediate reaction, we set time to 0 seconds.

Adding the time trigger and a speech playback modality in Adobe XD.

We can also add traditional triggers. In this example, we’ll add a tap trigger on the second element of the list. When the user taps on this element, the app will advance to the next screen. Combining both voice commands and touch commands is a great example of providing a better, more thoughtful user experience using multimodal design.

Adding a tap modality command in Adobe XD prototype mode.

Next, we want to illustrate how the user can pause and continue the experience within the app. Since we’re designing this app with the Amazon Echo in mind, we want to add a voice command like “Echo, pause.”

In this stage of the scenario, the video (illustrated with a still image) is paused.

To make the video continue, we’ll perform the same action by adding the voice command “Echo, continue.”

In this stage of the scenario, the video (illustrated with a still image) is no longer paused at the execution of the voice command.

This is a basic example of multimodal design using the voice trigger. Additional triggers include tapping, dragging, and using the keyboard or a gamepad to control the prototype.

It’s easy to fall into the trap of using triggers simply because they exist. To design a better user experience with multimodal design, designers will want to test and learn which interactions make the most sense, and at what time.

Multimodal Design and Mental Models

When designing using modalities, it’s important to remember that users have a preconceived set of expectations ( mental models) for how an interaction should occur. For example, most users expect a screen to move up when they scroll down on a trackpad or with a mouse scroll wheel.

Note that in many cases these mental models are still being formed. Shaking the phone is an example. It’s still an obscure interaction as some vendors use it to “undo” typing, while others use it to shuffle songs.

It’s important to be aware of these mental models when choosing which modalities to put into the product design. Using familiar modalities can enhance the user experience. Modalities which are still being formed could confuse users and degrade the experience.

WeChat uses shake to find people around you, or to identify a song.

Emerging Modalities: Conversation Design

Two modalities gaining a lot of traction are chatbots and voice user interfaces. Sometimes referred to as conversational user interfaces, the primary focus is text and speech interactions.

A chatbot can make use of an interface to receive inputs such as text, and is capable of showing graphics, links, maps, and conversational dialog. Some chatbots can take commands via voice, and then display the results as text or using synthesized voice.

Pure voice interactions are also emerging. Think of the expansion of Siri or Alexa to smart home devices where users aren’t typing anything, but instead are having full interactions with voice only. This is important for designers because almost every experience in conversation design is a multimodal one.

A great example is Lily from Maybe. A bot that teaches you Chinese (and other languages), and works on different channels. Conversations can take place on the app or by talking to the bot.

Lily teaches the user a new language through different channels, using different modalities.

Lily teaches the user a new language through different channels, using different modalities.

Summary

Using voice, touch, text, and taptic feedback, multimodal design combines different modalities for creating better user experiences. Computer-human and human-computer interactions can be combined to build unique product experiences.

Multimodal design also presents new opportunities and challenges for designers. Tools like Adobe XD make it easier to prototype products using various modalities, but there is an art and science to using them together.

Striking that perfect balance, combined with the emergence of new modalities, will challenge designers to raise the bar on improving user experiences.

Originally published at https://www.toptal.com.

Multimodal UX

Conversation Design

Frederik Goossens

Written by

Global Design Lead at HSBC, Conversational Banking • Managing UX, UI, VUI design • University of Cambridge: Executive MBA ’20 - ‘22 • Hong Kong based

Multimodal UX

VUX — VUI — MUI — MUX — CD

Frederik Goossens

Written by

Global Design Lead at HSBC, Conversational Banking • Managing UX, UI, VUI design • University of Cambridge: Executive MBA ’20 - ‘22 • Hong Kong based

Multimodal UX

VUX — VUI — MUI — MUX — CD

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store