Interaction Design Principles for Augmented Reality

Challenges and opportunities in designing natural interfaces for AR (and VR)

(I don’t represent Meta in this article. These thoughts are my own)

We have studied and understood interaction design principles very well over the last decade. A user interface exists primarily to serve an user by being delightful and easy to use. The seven fundamental principles of interaction design— Affordances, Feedback, Consistency, Non-destructive operations, Discoverability, Scalability and Reliability, are great guiding principles in designing great user flow.

In immersive AR (via a Head-mounted display), these principles are not only challenging to implement but provides a lot of opportunities for re-thinking the interaction principles in the context of AR (and VR where applicable).

  1. Perceived affordance —An object should naturally imply what actions it supports through its design and attributes. In AR, digital augmentations can take the form of simple information overlays or complex holograms. Field of view is a precious real estate in AR experiences where every item should have a purpose. It has to be meaningful from a user perspective to interact with an object the way its interaction is perceived. For instance, in a sculpting experience, a 3D model could be scaled, translated and manipulated upon. On the other hand, in an experience around a driving experience, where the user sits in a world scale model of a car, scaling and rotating objects does not make a lot of sense. So a meaningful affordance principle could be — Free form 3D objects can be interacted with and their properties changed while ‘glued on’ models cannot be.
  2. Feedback — The user should get feedback regarding an action they performed. Because AR uses the real world as a canvas, if an interaction includes a real world component, the feedback should take that into account. For instance, consider the instance of somebody moving and placing a hologram onto a table. A form of feedback can be a visual indicator like lighting the hologram and the table when the pairing happens. This tells the user that the digital object has locked onto a physical real world object. Other cases include during initialization where the AR headset maps the real world (SLAM). Check this video from Microsoft that provides great feedback when the hololens initializes to a new environment.
  3. Consistency and Standards — This will take more than several iterations to solve. There are no standards in terms of gestural interactions in AR. AR brings out an array of behaviors from different users in terms of how they navigate a three dimensional space. What is natural for one is not natural for another. This creates a challenge in defining consistent gestures and interactions.
  4. Discoverability — The intent here is giving users enough freedom to go on a journey. As part of this journey, they could discover what all they could do with the platform. There is a joy in discovering stuff and the product feels intimate after such a discovery process. Because navigation menus are not how we navigate in a AR world, discovery should happen as it happens in a real world. Introducing users to sets of interactions in a playful way may be the way to go. I am not yet sure what is a good discovery process for AR experiences, but this is a critical interaction principle.
  5. Scalability — Scalability refers to how well the interactions map to different environments (interactions working well for different screen sizes for example, in the context of web). This is one of the harder interactions to capture in AR because this not only depends on the accuracy of object tracking but how well gestural interactions work on objects with different sizes and form factors. For instance, grabbing a 3d object like a cube that is really small is not going to be same as grabbing a similar object with a different scale. Like in the real world, I might pick a small object with a pinch like gesture but grab a big object by their sides. So the scalability principle kind of falls apart the way it is defined.
  6. Reliability — Did I do what I intended? There are two kinds. False positives and false negatives. False positives are actions that happen when I don’t intend them to happen. For instance, I don’t want to move an object but it sticks to my hand and moves with it when my hand grazes by. False negatives are actions that don’t happen when I intend it to happen. For instance, I try to unsuccessfully grab an object that is meant to be grabbed. This interaction is one of the most frustrating aspects of AR/VR. This also relies on the underlying hardware and software algorithms working near flawlessly. There are some UX tricks one can play to improve the perceived reliability, but this remains a critical interaction.

These are great starting points to meditate upon. I think that as we see more apps and use cases, we will have enough data to start planning these interactions better. AR provides us an amazing opportunity to rethink interaction design in terms of what feels natural in a mixed-reality world. We just need to build a lot of experiences, collect data and learn. These are fun times.

If you like this article, consider recommending it. Thanks :)