Reality is Broken, So Design a New One.

An introduction to design in augmented reality.

Human imagination has never been satisfied with the real world. It’s too slow, too bland, and too hard to control. So, for as long as we can remember, we’ve been inventing new, crazier ones that let us live out our fantasies like walking with dinosaurs, warping space-time, and even falling in love with vampires. As incredible as these worlds are, we’ve been stuck experiencing them through words on a page or TV screens that let us peep from a distance and wonder what it might be like to live there. But that’s about to change in a big way. We’re finally seeing the beginnings of technologies that will let us open a door to these worlds and walk right in.

I’m talking about immersive technologies like Virtual Reality (VR) and Augmented Reality (AR) that, through products like Oculus Rift, Samsung Gear and Magic Leap, are close to becoming a real reality in our lives.

Here’s a quick reference on these technologies:

In a nutshell, immersive tech is new digital display medium that covers our entire field of view with a head-mounted display and, through nifty tricks like motion tracking, creates digital illusions that make us feel like we’re standing directly inside a computer generated world. The difference between Virtual Reality and Augmented Reality is not binary but an easy way to distinguish them is to think of VR as The Matrix, where you’re transported to an entirely new world and think of AR as Space Jam, where objects from another world add a bit of magic to our real world.

But the potential of immersive technologies goes beyond just letting us live out our imaginations. It turns out that simulating the real world is the most natural way for us to interact with computers and if you’ve seen the awesome way Tony Stark does his job, you’ve glimpsed at the future of how we’ll regularly work with computers and by that definition, almost every product we interact with.

Of course, there’s a dystopian view point, presented in the Matrix or in a, more recent short film, “Hyper-reality”, where we’re slaves to an artificial world that’s taken over and made our real world dull and dreary. But look beyond that dystopian view and you find a clever technical innovation that could free us from slouching over our desktops, burying our heads in our phones and maybe just create products easy enough for our grandparents to use.

That’s our vision for immersive technologies at Placenote. This blog explores the future of user-interface design in an immersive medium, with a focus on augmented reality. We believe immersive interfaces are the natural next step in the evolution of computing from the command line interface, to the mouse, to multi-touch screens and we’re excited about what this means for the software products we use everyday. The general design principles discussed here are applicable to the full spectrum of immersive tech but we’re most excited about augmented reality because it’s more social and fits better into our everyday lives by letting us multitask and interact with computers on the go. The benefits of AR v.s VR products might be a subject of a future post.

So let’s dive in and explore some of the basics of AR from the context of design. We won’t discuss technology, but rather investigate some advantages the medium presents for interface design. As far as technology goes, the ideal hardware device for AR is a pair of glasses with a transparent screen overlay . Examples of these are the Microsoft Hololens, Magic Leap or DAQRi helmet, but none of these devices have achieved the quality or price thats feasible for consumer adoption yet. Instead, the best current platform for AR development is the mobile phone camera. Apple and Google have released developer kits for AR app development (Apple ARKit and Google ARCore) that let developers use the phone camera as a window to view AR content. The format is not without its limitations (we’ll explore the limitations of mobile AR in a future post) but all things considered it’s the best option right now to get started with AR design.

So what makes AR interfaces so special? Every significant inflection point in the history of computing like personal computers, the internet and then mobile phones added an entirely new dimension to our interaction with computers.

• Personal Computers — At-home computing
• Internet — Connectedness
• Smartphones and LTE — Mobility

What is AR’s major paradigm shift? At face value, AR just seems like a better monitor that covers your entire field of view and lets you access digital content hands-free. But don’t confuse a heads-up display with augmented reality. The benefits of AR go far beyond just hands-free computing or natural interfaces.

The biggest advantage of AR is that it makes computing ubiquitous. Today, when you walk into a store and see something you like, you take your phone out to compare its price on Amazon. Or, if you buy a new coffee maker, you might look up instructions on how to replace the filter. Each time we pull out our cell phones to search for something we’re explicitly engaging a computer and explaining our situation to it in order to get what we need. This, while we may not notice, is a fairly cumbersome process.

Each major innovation in computing has gradually peeled away the barriers to our interaction with computers by making it easier, more useful and more fun to use. The promise of AR devices is an always on, contextually aware computing device that can tell you what you need to know without searching for it. This isn’t limited to visuals. If you haven’t yet, please do yourself a favor and watch the movie “Her” by Spike Jonze. It beautifully illustrates a potential future where always-on voice interfaces are the primary mode of interaction with computers. The spectrum of immersive technology goes beyond visual effects and when thinking about design for augmented reality we should consider all the ways we can create always-on, contextually aware, natural interfaces for people to interact with their world.

Thinking like an augmented reality designer

When thinking about designing apps, it’s useful to have a high-level categorization of use-cases that AR enables. At Facebook’s F8 conference earlier in 2017, Mark Zuckerberg defined three ways AR could be implemented in an app.

1. Add digital objects to the real world. (e.g. play chess on a virtual chess board)
2. Display information on a real object or location (e.g. augmented reality guided tour of the colosseum)
3. Enhance existing objects (e.g. Share photos with Snapchat face filters)

I like this categorization, but I want to build on it by first exploring the value propositions of AR to users, before discussing options for implementation for developers.

At a high-level I think there are four main value propositions of augmented reality apps to users. I’ve added a sample user story beside each value proposition.

1. Visualization for comprehension:
   e.g. Visualize what a new IKEA couch will look like in your living room before buying it.
2. Information in context:
   e.g. See instructions on how to replace a printer’s toner cartridge directly overlaid on the printer.
3. Immersive experiences:
   e.g. Walk beside dinosaurs to experience what life must have been like in jurassic ages.
4. Natural interfaces:
   e.g. View and collaboratively edit an architectural model of a building on a table top or directly on site.

The Universal User Interface

The science of human computer interaction (HCI) explores how humans, with their senses and motor skills can interact with machines in the simplest and most efficient ways. There is a lot to be said about good user interfaces and if you’ve pulled on a door that was supposed to be pushed, you know that even something as simple as a door can benefit from a good user interface. Don Norman, the king of functional design has a lot to say about this topic in his book, The Design of Everyday Things. The idea is that the more intuitive a product is, the less cognitive stress is felt by its users and as a result, they love the product and use it more often.

I think, by far, the greatest promise of AR is in its potential to be a universal user interface to the world. Today we use all kinds of different methods to help us navigate the objects and spaces we come across. Printed markings on knobs tell you how to use a dishwasher, huge billboards tell us how we ought to be spending our money, TV screens tell us when the next flight is and mobile touch screens let us navigate to our destination. We’re visual creatures and we’ve invented thousands of ways to communicate through imagery. In AR, all these visual media become redundant. AR could give us a simple interface to navigate, explore and understand anything in our world in a very intuitive way.

Re-invent all design or Iterate?

I hear a lot about AR native design and how the principles of 2D design need to be thrown out to rethink the way things will be done in AR. I think that’s a bit drastic. We haven’t even begun our transition phase into the 3D paradigm so I would rather use the principles of 2D design to draw analogies and figure out how to position ourselves best for 3D interfaces.

AR changes the paradigm of user interfaces from a flat two dimensional screen to a 3D world that can be explored by literally turning your head or walking through it. The concept of browsing might have a totally different meaning in AR. Speaking of browsing, a lot of concepts in 2D GUI’s like files, folders, navigation, browsing were created as skeuomorphisms to help people relate to an unnatural 2D interface by using visual analogies from the real world. It seems almost as though our work this time around is easier. Perhaps we simply need to take all those skeuomorphisms and turn them back into the real world interactions they mimic to build our 3D interfaces.

Science Fiction likes to show augmented reality interfaces as glowing complex 3D displays filled with graphs and charts. These are built for mutants. Interfaces built for regular people need to be simple and intuitive to be adopted. The future of AR computing is simple and it’s up to designers to take us there.

Footnote:

In future posts we’ll explore specific examples of design in mobile AR and look as case studies of apps built in AR to understand how to build better products and avoid common pitfalls in AR design. If you’d like us to cover a specific topic you’re looking to learn about, drop us a note with any suggestions. We’d love to hear from you. If you’d like to write for us, please do send us a note as well and we’d love to hear your story.

Placenote is an app for iOS that lets anyone create simple augmented reality interfaces to physical spaces and objects. Placenote is built on a persistent AR engine that enables digital content to be permanently saved in physical locations and shared through the app.

To learn more about Placenote, visit us at https://placenote.com.