Why Mobile AR Matters
Though it might be tempting to view it as “just an interim technology”, handheld Augmented Reality is actually here to stay. Let’s have a closer look.
It’s 2018, and the race is on between tech giants to decide who will control what you see every time you open your eyes. Augmented Reality— the technology that seamlessly inserts computer generated content into your surroundings— has been championed as the next computing platform, and it’s finally hitting the shelves.
Yes, say goodbye to touchscreens, your next tech gadget will be a stylish pair of AR glasses. You heard it right, Augmented Reality is finally there! Well, kind of. Actually, you can forget high tech goggles. For now, Augmented Reality is coming… to your smartphone. Wait, don’t leave yet, it’s actually pretty exciting.
Last year, Apple introduced ARKit; the first truly robust and mass market Augmented Reality solution. As Tim Cook was quick to boast, ARKit turned the iPhone and iPad into: “the biggest AR platform in the world almost overnight”. Apple’s announcement was quickly followed by Google’s own release, a similar platform for Android called ARCore.
In both cases, the AR effect is obtained by blending 3D content in real-time over images from your device’s camera. Clever software matches the angle of the virtual camera to the position of the device, and estimates the illumination of the scene to realistically light the object.
“I see AR as being profound. I think AR has the ability to amplify human performance instead of isolating humans.” — Tim Cook, CEO of Apple
But are smartphones really capable of delivering on the promises of AR, or is this just a temporary solution, rushed to market to beat the competition? As every tech giant is working on their own wearable version of Augmented Reality, is there anything intrinsically valuable to be found in the modest handheld version?
What the heck happened to Augmented Reality?
If you roamed the corridors of the University of Utah in the late 1960’s, you might have come across the disturbing scene below.
That creepy looking contraption is not some antique mind control device, or a prop from a Terry Gilliam movie, but the very first Head Mounted Display (HMD). It was created in 1968 by computer graphics pioneer Ivan Sutherland and his student Bob Sproul. Sutherland called it “the ultimate display”, but it is now remembered by its more threatening nickname: “the sword of Damocles”.
“I predict that AR will involve the greatest leap in the nature of human/computer interactions since the mouse and GUI.” — Michael Abrash, Chief Scientist at Oculus
Sutherland’s pioneering —if slightly scary— invention dates back exactly fifty years… the same year Douglas Engelbart demonstrated the computer mouse for the first time!
With the rate of progress we see in computer technology, you’d think we’d all be running around with digital contact lenses by now. Obviously, this is not the case, so what happened?
AR glasses are really hard to build
A typical sighted person gets 80% of their sensory impressions through vision alone… or so ZEISS’ website claims. Whatever the actual number, our eyes and brain are really good at telling what’s what. When engineers are trying to fool our visual cortex into believing something is there that really isn’t, they are fighting against millions of years of evolution, leading up to the advanced visual system bringing these very words to your attention right now.
Therefore, overlaying realistic digital images on a person’s view of the physical world is really, really, really hard. Doing so without making the wearer look like a massive cyber-dork is even harder.
Image by Facebook
Though most experts seem to agree that glasses are the gold standard for Augmented Reality, they are not going to happen tomorrow (scroll further down for a word on current head mounted displays). You don’t have to believe me, just take it from the man himself:
“I think everyone would basically agree that we do not have the science or technology today to build the AR glasses that we want.” — Mark Zuckerberg
It looks like our handheld devices are here to stay for at least a little while longer, and it might be a few decades before Augmented Reality glasses become as ubiquitous as smartphones are today.
So, even if mobile AR is “just an interim technology”, the transition period is going to be pretty long. That means the resources you put today into mastering mobile AR will not be wasted. Which brings us to our next point…
Mobile AR is its own medium
With every new medium, we see the same story play out; we don’t take time to look at that thing we created with fresh eyes. We keep re-using techniques, formats, and principles that worked in the past but may be inadequate for this new environment. This creates awkward hybrids that compare poorly to the status quo, and hides the true potential of the new medium.
We are only beginners poking and prodding our creations, not quite seeing them for what they are, and always unconsciously trying to understand them in terms of the old and familiar. — Nicholas Gessler
Take cinema for example: many early movies were just theatre plays captured on film, with a fixed camera and no editing. It took true experimenters like French illusionist George Méliès to take the real measure of what cinema could be. When Méliès started working with a camera — though he had already had a prolific career in theatre — he didn’t simply film actors on a stage. He immediately started experimenting, inventing new tools, and creating special effects that were unique to the medium.
This is a lesson for anyone getting into designing for mobile AR (or any kind of emerging medium for that matter). We don’t know yet what will or will not work. Don’t try to force the old world into the new one. Get familiar with the tool, experiment, try new things, find what is unique to this platform.
But with mobile AR, we have an extra layer of confusion. It is coming from the expectations of immersion that AR (and VR) headsets have set for people, but which a mobile device cannot deliver. This is why we have to remind ourselves of the following hard truth.
Mobile AR is not AR
No matter how much you wish you were designing content for a pair of AR glasses that doesn’t exist yet, your ARKit/ARCore app is actually going to be running on a mobile device. And here’s another secret: mobile AR is not AR… not really. In fact, it would be more accurate to talk about “camera applications” or “lenses”. See The Camera As A Platform by Allison Wood for great arguments on that point.
Your users are only going to see a tiny part of your carefully crafted 3D environment at any given time, so if you are designing a fully immersive experience, you’re designing for the wrong platform. This also goes for any scenario that demands that people hold their phone up in front of their face for an unreasonable amount of time.
Image byTechCrunch/Youtube
Mobile AR doesn’t have to be a stepping stone to anything to be valuable in its own right. If we want to deliver compelling experiences that users will enjoy and come back to, we need to invent new design principles, visual language, and input methods that are adapted to the platform. In other words, we have to accept and design for what the medium is, not what we wish it would be.
In time, these new ideas will become the bedrock of mobile AR design; a set of accepted design principles and best practices for the medium… that the next generation will have to tear down again when it’s time to move to the next thing.
How can we begin to understand what this new medium is good for? Well…
Experiment!
Before we start writing rule books, we have to understand the medium and explore what’s possible. This is why the work experimental artists are doing in mobile AR is so crucial.
I think we’re going to see some rapid evolution as artists experiment directly in AR. The symbiosis between the artist and the toolmaker is going to be an incredibly important relationship over the next few years. — Matt Miesnieks
Multi-talented artists like Zach Lieberman or Isaac Cohen are the Méliès of the modern era. They are the ones inventing their own tools and getting an intuitive understanding of the medium.
We need more pioneering artists like them to creatively research what can be done in the AR space, but also designers to prototype meaningful use cases, and engineers to push the limits of what is technically feasible. We must work across disciplines to really discover the potential and understand the boundaries of this new territory.
It will take some time before we truly understand what can be done, how to overcome technical limitations, and for the ecosystem to reach maturity… but that doesn’t mean there aren’t some great ideas and concepts out there already! Let’s have a quick look at some interesting use cases and interaction ideas from the recent crop of AR concepts doing the rounds online.
Calibration as a game
Pokemon Go is often credited for re-igniting the excitement for mobile AR, which is strange, since until recently, it actually had very little to do with AR. That was fixed in the recent update that introduced the “AR+” feature.
The game designers used a clever trick, to get the user to scan the environment: before catching a wild Pokemon, you have to search for it through the bushes. This makes the user move their phone around, which is necessary for a stable tracking.
On the negative side, AR+ actually makes pokemons harder to catch since you have to keep the creature in the frame AND aim the pokeball at the same time (in the non-AR mode, you only have to throw). Is that an acceptable tradeoff for a more immersive experience? Players will decide.
Key learnings
- Hide calibration steps by justifying them through gameplay or storytelling elements.
- If the AR feature in your app makes it less usable, ask yourself it it’s worth including.
Personified navigation
Hotstepper is “a confident dude who, when he’s not dancing, will walk you to any location you need to go”. In this app, you enter your destination, and the little guy marches in that direction as if leading some kind of naked parade, as you follow… with or without clothes on, the choice is yours.
Although the app is not exactly practical in its current state, it shows the kind of out-of-the-box thinking mobile AR calls for: does navigation in the physical world really need arrows, street names, and turn-by-turn instructions? Why not have people follow a little naked dancing dude with a majorette hat instead?
Key learning
Mobile AR opens the doors for playful and unconventional solutions to seemingly well solved problems (like navigation).
AR as a feature
This concept Isil Uzum posted on Dribble shows the potential of mobile AR when it is cleverly integrated within an existing app.
Even though is would be challenging to make it work seamlessly using the current technology, leaving video instructions for guests right where they need it is a total no-brainer. As positional tracking improves and content can be persistently mapped to a space, no doubt this type of interaction will become ubiquitous.
Key learning
Mobile AR can work extremely well as a feature within an existing app.
Augmented Glimpse
This interaction is part of another navigation app called ARCity. Based on the angle of the phone, the app will either show a traditional map, or switch to an augmented view of the street, overlayed with 3D arrows showing you the way. This is another interesting way to implement AR as a feature that can be accessed in a very intuitive manner.
Key learning
Contextually switching from a traditional UI to AR can be very powerful.
Side note: on current HMDs
As impressive as the current generation of head mounted display are, they still fall very very short on many points (price, style, field of view, usability…) It will take serious breakthroughs before we get the kind of technology that makes you forget you’re wearing glasses and truly blends digital content seamlessly in the world around you.
For an in depth technical look at the current limitations of AR glasses, and what it will take to bring them to the mass market, I recommend this excellent article by @mattmiesnieks.
That said, by all means, try them on! You’ll learn a thing or two about AR and probably get excited about the future. If after that you’re still in the mood for the kind of Augmented Reality that can impact people’s lives today, come have a chat with us at NEEEU. We have a few ideas of our own about Mobile AR and the role it will play in the ongoing spatial transformation of technology and UX. We’d love to share them with you.
What’s next?
In our next post, we’ll cover some practical tips about designing for mobile AR, dig deeper into ARKit’s pros and cons, and share some insights from our own experiments at NEEEU. Stay tuned!
🐸 Raphaël de Courville wrote this article. He is also a co-founder of NEEEU. You can find him on Twitter at @sableRaph or contact him directly at r@neu.io.
🤹🏻 Identifying which spatial technology has the potential to become a great product or service is a full-time job. It’s also not your job. Luckily for you, that’s exactly what we love doing at NEEEU. Wanna create delightful services and products that live in the real world? Get in touch at hello@neu.io
