Multi-Modal Interactions for 3D UX Design

Rajat Kumar Gupta
Mar 14, 2019 · 6 min read
Image for post
Image for post
brickvisualvr

Introduction

This article aims at developing the thinking approach towards any immersive experience (i.e Augmented Reality, Virtual Reality or Mixed Reality) that you are planning to develop. The design principles that I will be covering here should prove useful while developing any immersive experience.

Multi-Modal Interactions

The moment one thinks of developing any experience in XR, one out of the five senses of the developer gets activated first i.e sight. No doubt it is the most important aspect, but overloading one sensory system is a bad idea. Users tend to perform better with multi-sensory UI’s namely speech, gaze and gestures could be a good blend. These give a pretty good connection with the kind of interaction that users want while interacting with objects in the virtual world. The idea is to make an intuitive experience. It’s a good thing that XR is multi-modal by nature which makes it so compelling. Let’s discuss some common principles that are being incorporated into developing immersive experiences.

The 3 input mechanisms that this article will cover are the following —

  • Gaze
  • Gestures
  • Voice commands
  • 6-dof controller()
  • Gloves and direct interaction eg. Leap Motion()

The various feedback mechanisms are the following —

  • Visual Feedback
  • Audio Feedback
  • Haptic Feedback ()

Gaze

Gaze is a very powerful UI. Just start looking at any random person for long. There are various outcomes that can happen —

  • The person might just ignore and walk away
  • The person might get curious and come towards you
  • The person might start or stop doing an action

These outcomes if incorporated into your experience might prove beneficial in terms of making a convincing interface for your user. Thus bridging the gap between the real and the virtual world. However, gaze is well known for its cursor like capabilities in the world of immersive experiences. Below is one of the sample demos from the official Google VR repository

Image for post
Image for post
The white dot is the cursor
Image for post
Image for post
Gaze on an object and it gets highlighted. Also, the cursor expands.

This particular experience gives visual feedback to the user that there is an object to interact with and the element is active. A few Gaze guidelines —

  • An object of interest in the environment that you are gazing at should be kept at a minimum distance of 0.5m as recommended by Oculus. 1 meter is a comfortable distance for menus and GUIs that users may focus on for extended periods of time.
  • The optimal gaze angle is 10–20 degrees below the horizon because the user generally tends to tilt the head a bit downwards during the experiences.
  • Keeping the subject in focus. If the user loses track of the subject it is the job of the developer to draw back his or her gaze back to the subject by providing audio cues or visual cues. Check out the Google ARCore Elements app that gives subtle cues and this section of the official Augmented Reality Design Guidelines by Google.

Gestures

Gestures serve as a simple technique for users to interact with virtual objects in mixed reality. The idea behind this input mechanism is that users tend to move their hands a lot while they are participating in an immersive experience. It just happens intuitively and thus there are several meaningful gestures that are being incorporated like the famous “air tap” gesture for the Hololens(image below)

Image for post
Image for post
The Air Tap gesture for selection/deselection of objects in Hololens

One thing to keep in mind while integrating a gesture is the arm fatigue of the user. Repeated gesture input is tiring for the user. Give the user a break! No seriously, incorporating short breaks or offering a blend of speech and gesture inputs could go a long way in increasing user engagement for your app.

Image for post
Image for post
Stop making me air tap all the time!

A few Gestures guidelines —

  • Raising the arm so high that the elbow is not in contact with the user can be tiring.
  • People lose track of their real-world physical surroundings and poor hand gestures can result in them hitting a wall and eventually getting involved in a bad accident.
  • Hands that visually look realistic can be creepy because it gives the user a feeling of being inside someone else’s body. Therefore breaking the immersion

Voice

Voice(speech) interaction can add efficiency to the experience. Voice UI in XR is as simple as — See it, say it. This means that if you are gazing at an object and want to interact with that object, you give the voice commands instantly. For eg) Gaze at a dog, say “jump” and it jumps. You won’t say “selected dog jump”. Keeping speech UIs simple with short phrases is one key aspect of speech UI design principles.

Voice can be imperfect in many ways. For instance, if you want the speaker’s volume to increase you say “louder” but you can’t say “a little louder” because it is hard to quantify. Sometimes a voice system incorrectly hears a command or fails to hear a command. Recovering from such errors is a challenge in any interface. Besides voice may not be socially acceptable in public places.

Image for post
Image for post
Source: Microsoft docs

A few Voice guidelines —

  • Give users a proper logging output of their voice while they are taking part in the experience. It is called user input logging. Check out this video
  • Use simple vocabulary in your voice commands.
  • Make sure there is a way to undo the voice command so as to prevent any unintended action due to some other person speaking.
  • Test your app with different accents.

Conclusion

The guidelines covered in this article are just one topic of the various UX design principles that can be applied to this exciting new field. Will be coming up with more articles soon!

Additional Resources

Swevens

Disrupt and revolutionize world with state of the art…

Rajat Kumar Gupta

Written by

Software Developer | AR/VR Prototyper | Writer, AR/VR Journey Magazine | 117.8K+ views on Quora | @knightcube | www.rajatkumargupta.com

Swevens

Swevens

Disrupt and revolutionize world with state of the art products and solutions using AR, VR, AI & IoT. We strive to take humanity to new heights.

Rajat Kumar Gupta

Written by

Software Developer | AR/VR Prototyper | Writer, AR/VR Journey Magazine | 117.8K+ views on Quora | @knightcube | www.rajatkumargupta.com

Swevens

Swevens

Disrupt and revolutionize world with state of the art products and solutions using AR, VR, AI & IoT. We strive to take humanity to new heights.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store