Practical Design Considerations for Augmented Reality Apps on Mobile Phones

Published in

Placenote Blog

9 min readNov 29, 2017

I believe that augmented reality is the future of UX for almost all computing experiences. In my earlier post, I discussed how immersive technologies break the cognitive barriers of human-computer interaction and let us interact with technology in the most natural way we’re used to. We also looked at the four high-level advantages of AR interfaces for users as being:

Visualization — e.g. Visualize what a new IKEA couch will look like in your living room before buying it.
Contextual information —e.g. See instructions on how to replace a toner cartridge directly overlaid on the printer.
Immersive experiences — e.g. Walk beside dinosaurs to experience what life must have been like in jurassic ages.
Natural interfaces — e.g. Collaboratively edit an architectural model on a table top or directly on site.

But there’s a problem with this approach. So far, we’ve primarily focused on augmented reality in its fully mature form. In fact, I’ve found that most designers and developers are mesmerized by the promise of AR in a world where everyone is wearing a pair of lightweight and fashionable AR glasses that have completely replaced laptops and cellphones. The reality is a bit more humble. It’s clear from the progress of AR devices like the Microsoft Hololens, Meta and MagicLeap, we’re still a few (and maybe more) years away from mass consumer adoption of AR glasses.

In the interim, the best alternative is already in our pockets. Apple and Google have recently launched tools (Apple ARKit and Google ARCore) to help developers create mobile AR apps that use the phone’s camera as a window to the augmented world. This is a great hack, but it presents unique problems for UX design in AR. Let’s explore some of those problems here:

It’s not always on

Unlike glasses, mobile phones aren’t always on. They’re usually in our pockets. Mobile AR demands a behavioral change from users. No one is used to holding their phone up to eye level when using an app except when taking photos. Try holding your phone up with one hand and time yourself. It’s uncomfortable to do it continuously for more than 30 seconds and it’s not easy to hold the phone vertically and click the touch screen with the same hand.

If that wasn’t painful enough, this varies based on the phone. An iPhone 7 is easier to hold than an iPhone 7 plus, which is way easier to hold up than an iPad Pro. In fact on an iPad, users will use both hands to hold the device, so maybe you should minimize touch interactions. Let’s consider a contextual limitation. If your AR app involves interaction at a grocery store, your users will probably be pushing a shopping cart with one hand, if not both. That makes it a lot harder to drive adoption for your app, purely because of the awkward gymnastics needed to use your app.

What can you do about it?

We learnt these lessons the hard way at Placenote. We created guided tours for museums in AR using dedicated Google Tango devices stocked at the museum’s front desk. Users would pick up tablet at reception and walk around the museum with it. We expected users to carry these phones with them for at least 15 minutes. Unfortunately these devices were heavy and some people would just leave the tablets on the museum floor when they got tired of them. We had to install lanyards and handles to make it easier to view AR content without extending your hands too much.

The best way to mitigate these issues is to acknowledge them and find work arounds. Design your UX for short and quick AR engagement and piggy-back on familiar behaviors like capturing a video while viewing your AR content. This doesn’t mean users can’t engage with your app for long. It just means you need to balance the ratio of 2D and 3D interactions with your content based on what is most comfortable for users. Pokemon Go does this really well (see image below)

It’s not immersive

The main thing to remember is that when viewing AR content through a phone, you’re looking at content through a tiny window in your hand. So, while it’s tempting to think of the entire world around you as an available canvas for content, the truth is that doing this will frustrate your users. Imagine viewing a huge dinosaur with your phone. Panning your phone around just to view a single piece of content is a very uncomfortable viewing experience for users.

This also relates to users holding their phones up to eye level. Since you’re limited to interactions with one hand, creating content at awkward angles or viewing distances to the user is also going to annoy your users.

What can you do about it?

Take into account the appropriate viewing distance of your content and design your 3D content to fit comfortably in your field-of-view. Don’t add content too high or too low below the user and don’t make people read text that’s too far away or too small to view through the screen.

It’s not a natural interface.

In a traditional 2D app, there’s no ambiguity about what the goal of the touch screen is. It’s a set of UI panels and clicking on buttons activates actions or navigation. The only way to interact with the UI is through touch. This has been the way mobile UIs have been designed for the last 10 years.

The most interesting and challenging user interface paradigm designers need to consider when designing AR apps is that the phone screen serves 3 purposes in mobile AR. It’s a UI for 2D interactions, a viewing window to the 3D world and a 3D cursor.

For example, you might want to select a couch from a 2D grid of options (2D UI), position the couch by pointing your phone to a drop point (cursor) and then walking around the couch to view it from all angles (viewing window). This means you need to design a UX that clearly signals to the user what the current function of the screen is.

What can you do about it?

When possible try to consciously minimize the functions of the screen. Remove all 2D interactions unless absolutely neccessary (like text input) and try to avoid using your phone as a cursor and viewing window simultaneously.

If you need to use all three, do it deliberately. A good example of this is the iPhone app “World Brush”. The screenshots below show how they use the touch screen and the phone’s motion simultaneously to draw objects. They have a dedicated mode where users can use just the phone’s motion to paint, just like spray painting.

The technology is not totally robust yet.

Augmented reality is a digital illusion that is created using some nifty technology on your phone. Without going into too much detail, the camera and the motion sensor on your phone work together to track how the phone is moving and understand its environment. The camera observes features in the environment like edges and corners of objects to estimate its own motion and can also detect horizontal planes in order to place a digital object on a flat surface. This is a technological marvel but it’s not without its limitations. It’s possible in some circumstances to “break tracking” and arbitrarily throw off the position of your 3D content.

The factors that lead to this are rapid movement or poor visibility in the camera like using the app in the dark or pointing your phone at flat textureless walls.

What can you do about it?

Don’t create experiences where your user might tend to move their phone too drastically because it may break tracking. For example, let’s say you’re try to build a game like fruit ninja in AR, where you’re swinging your phone around like a sword to slice virtual fruits. While this sounds insanely fun, it might be a bad idea because depending on the environment it might break tracking. If your app lets people place virtual paintings on walls, don’t let your users get too close to the wall because it might break tracking if the camera can see nothing but a plain wall.

General points to remember.

In addition to the points above, there are two more important factors that, I think, should be considered when designing an mobile AR experience.

AR is an illusion. Preserve it as much as you can.

This is more of a general best practice for AR design. The problem with designing for AR is that you don’t exactly know how it will look for each user. The surroundings become part of your UI. Think about the different types of surroundings your app will be used in. If the environment is very busy or lacks flat open spaces, it might be a problem. This is especially noticeable when AR content shows through something that should be in front of it spatially. It looks terrible, totally breaks the illusion and diminishes the user experience a lot. You may have very little control over this but try to cognizant of it throughout the design process.

Does your app need to be an AR app?

Like I mentioned before, the UX problems with mobile AR mean that the bar for engagement in AR apps is much higher than regular mobile apps. Looking at the four value propositions of AR — (1) visualization, (2) contextual information, (3) immersive experiences and (4) natural interfaces, I think that, in most situations, the latter two are not the best fit for mobile AR since the medium is neither immersive nor a natural interface. That leaves (1) and (2) as possible advantages of AR to your app.

It’s tempting to incorporate AR as a wow-factor element but its actually very difficult to accomplish that on mobile AR without (2) and (3). A good way to examine whether AR is essential for your app is think about what you are augmenting, why you’re augmenting it and whether the same goal could be accomplished in 2D. Let’s look at some examples below.

Augmenting nothing — e.g. table top games built mainly for an immersive experience
Augmenting an object — e.g. face filters, augmented wine labels, Nike shoes ads
Augmenting a place — e.g. Pokemon go, guided museum tours, visualizing furniture in your living room

When your app augments nothing, or in other words, is contextually unrelated to the surroundings, it’s probably best as a VR experience rather than an AR experience. There are cheaper VR headsets that are a lot more immersive than anything possible in AR right now.

When building an app that needs AR for visualization, are you visualizing something because you need to view it in context like a virtual couch in a real living room or are you just looking to visualize it in 3D? Sometimes a photograph or a 3D model on a screen works a lot better.

Even if your app does augment something, you need to be aware of the context in which users will engage with your content. For example, Yelp has a AR mode in their app called Yelp Monocle. It lets you hold you phone up and see all the restaurants around you in their correct geographic locations marked with virtual pins. At first glance this may seem like a good idea but it hasn’t been adopted by users. Why? When you think about how you search for restaurants in an area, you just want to know what’s generally around you and you care mostly about reviews and cuisines. You don’t really care about the accurate spatial location of that restaurant and you’re probably ok with walking a bit more for better food. So in this case a map view is actually a better UX alternative.

Conclusion

Augmented reality will be the future of how we interact with computers. But, right now, we’re in a transition phase where mobile phones are the best and most widely available AR platform out there. Design your experience wth a firm grounding in the reality of mobile UX for the best chance at success with your users.