What’s Next for AR?
AR went mainstream in 2017; and it happened without most of us noticing.
[Transcript of my talk at AWE 2018 https://augmentedworldexpo.com/sessions/whats-next-for-ar/]
Good afternoon, everyone. I’m delighted to be with you today, and as always, humbled and grateful to be invited to speak at AWE. Given our unique purview at Unity, we see a lot from developers using our platform to build the full range of immersive content, and are in close touch with partners making the hardware and infrastructure that will power the AR future. So I’m excited to share my thoughts on what lies ahead. But before we do that… let’s take a quick look at where we are.
Here We AR
I think it’s safe to say that the hype around immersive technology has moved fully from VR to AR in the last year. Many believe that VR has stalled, stuck in the “gap of disappointment,” “trough of disillusionment” — whatever you prefer to call it — and still has a long, slow climb up the “slope of enlightenment” before reaching market acceptance. Meantime, entrepreneurs, investors and customers have turned their attention to AR, in the expectation that it will get broader uptake, faster.
With that as a backdrop, now we’re faced with a question: will augmented reality, which has actually been around in one form or another about as long as VR, and is undergoing something of a renaissance, also fall into its own chasm? What do you think? Are you feeling a bit worried that AR, too, could face huge hurdles to mass adoption? Are you wondering what it’s going to take to jump that gap? Dying to know when AR is going to hit the big time? C’mon, show of hands. Who here wants to know what it’s going to take for AR to go mainstream?
Well I’m here to tell you: we’re already there. AR went mainstream in 2017; and it happened without most of us noticing.
You Say You Want a Revolution
We’re into immersive technology because of the dream that it’s a revolutionary new medium — one that will change the way we work, play and communicate; that the computer interface will escape the tyranny of the rectangle, so that information is all around us, displayed in friendly, familiar, real-world metaphors; and that through these technologies we will learn more, share more moments from our lives, be more empathetic, more productive, and have more fun. And we expect that the immersive world that seems so tantalizingly close to becoming a reality will make our world look very different from what came before.
The thing about revolutions is, when you look at them in hindsight they can appear like massive change that happened all at once — the result of a single, identifiable event. But in reality, revolutions are a series of small changes, unremarkable on their own, that eventually reach a tipping point. Typically, the conditions for a revolution are set long before the outcomes become obvious. This has been true with political revolutions, scientific revolutions, and technological revolutions.
And it’s true for XR. The history of developments that led up to where we are today is too long, with too many examples to cite here, and it’s been talked about a lot at conferences like these. So instead of rehashing all of it, let’s take a look at what’s happened in just the last two years.
Arguably, AR’s watershed moment was the release of Pokémon Go in the summer of 2016. This location-based game, based on one of the world’s most beloved and recognized IPs, uses the camera to bring virtual creatures into the real world. And while the AR technology was rudimentary — just sprites overlaid on the camera image — the blend of digital content with world around us made for thrilling game play, and it generated massive excitement. Pokémon Go was and remains one of the top-grossing, fastest-growing games ever.
The success of Pokémon Go unleashed a flurry of activity that brought together trends in hardware and software, including renewed interest in Vuforia’s toolkit, and lots of experimentation by companies like Snap and Facebook around combining fun virtual content with your face and your environment. This broadened the appeal of AR beyond gaming to other use cases in social, brand experience and more, with the camera at the center of the action… and began to educate hundreds of millions of consumers about the wonders of AR.
But everything really, truly changed last summer with Apple’s release of ARKit. It was the first spatial AR on mobile, with real 3D objects anchored to the surrounding environment, now available for the masses. And with Google’s ARCore shipping earlier this year, we are now on track to have a BILLION AR-capable devices in 2018. In its wake, there has been a rush of development by companies large and small. Facebook and Snap have moved beyond filters and stickers, to lenses, 3D environments and objects. A lot of the energy and attention that had been on VR quickly moved to AR during this period, drawn by the promise of the scale of mobile AR, and the availability of immersive content without the need for a new, alien and — let’s face it — downright goofy apparatus up on your face.
So, here we are. The stage is set. The necessary conditions are in place, with true, if a little basic, AR on a billion devices. But is that sufficient? Obviously not. Without compelling content and useful market applications, this is all just cool technology.
The reason that AR went mainstream in 2017 is that real applications finally arrived, at scale. Let’s take a look at a few of them, spanning enterprise and consumer, across mobile handsets and dedicated headsets.
I think a lot of people got really excited with the release of ARKit, and then got quickly deflated when the first apps came out. For the second half of last year, we mostly saw demos, like cool virtual sculptures placed in public parks, and simple game levels — not full games. It was all very cool, but only for about a minute. So there was a perception at the end of last year, that has spilled into 2018, that ARKit and ARCore were just for demos and nothing more.
Well, that’s because it takes time to develop great apps, games in particular. Now, several months into the life cycles of ARKit and ARCore, there are literally hundreds of mobile AR games deployed through the major app stores. Everything from racing games, to sports, to shooters, to puzzles and scavenger hunts. (We know this because Unity’s analytics platform tracks devices and capabilities such as when ARKit and ARCore-based apps are installed.) Here’s just a partial list of the great AR games that have been released.
A couple of other notable Made with Unity titles:
- Walking Dead: Our World by Next Games (not yet released- but will be avail on both ARKit and ARCore) https://www.thewalkingdeadourworld.com/
- Jurassic World Alive by Ludia Inc. (avail on both ARKit and ARCore devices) https://venturebeat.com/2018/03/10/jurassic-world-alive-ar-game-is-pokemon-go-with-dinosaurs/
So, gaming, the bellwether of health in the mobile app ecosystem, is now very well represented with hundreds of AR titles. It’s a bit early to judge the results based on daily active users or revenue, because many of these titles only shipped recently. But I assume we’re going to see some amazing AR game play, and commercial success, this year.
And let’s not forget that sometime in the near future, Niantic’s newest location-based AR game, Harry Potter: Wizards Unite is going to ship.
Shifting gears from consumer gaming and entertainment to the enterprise, it’s a different picture.
First, on the hardware side, we see a variety of devices in use, including headsets like the Hololens, and tablets as well as phones. There’s a lot less sensitivity around price and form factor when you get into an enterprise setting. A company will shell out three to five grand for a headset if it will make their workers more productive; and while most folks wouldn’t want to walk around city streets with a Hololens on their head, they wouldn’t have a problem doing it in the design lab if it’s helping them do their job. And tablets provide more visual real estate than phones, so it’s worth hefting them to look at a virtual manual that provides instructions on assembly or repair.
Here are a few great examples from the enterprise world. Microsoft partner and Unity developer Valorem, based in the US and Germany, is developing AR enterprise applications that are a natural extension of their existing practice of solving problems for customers in automotive, architecture and other industries. Here’s a tire design solution for the Hololens, that shows a product virtually before it’s built.
Valorem then also used that same design solution in a training and sales presentation that incorporates their HoloBeam AR telepresence system using HD depth cameras, WebRTC to transmit the data, Vuforia model targets, sophisticated polygon reduction, and it plays back in Hololens or 2D displays.
The Greatest Stories Ever Told
Let’s touch on one more application domain. The possibilities for AR in sales, marketing, advertising and e-commerce are limitless. Immersive technology is arguably the most powerful storytelling medium, ever. And that’s what marketers do all day long: they tell stories: stories about how their product will make you smarter, better looking, thinner, sexier, richer; stories about how their company has the best people, the best technology, the most trusted brand; stories about how your life will be better if you just use their stuff. With AR, we have a new and amazing tool in the storytelling toolbox, to reach, engage and directly sell, all within a single experience.
You’ve probably seen IKEA Place, the app lets you try furniture out virtually before you buy it. And Amazon AR View, where you can see select products in your home, from within Amazon’s app. Now imagine that instead of a bespoke shopping app that delivers the experience, that something like this shows up in any mobile app dynamically as sponsored experience or ad. Our friends at RYOT Studio, a division of OATH, developed this Home Depot shopping experience for Yahoo! Mail. The AR experience launched from a native ad within Yahoo! Mail: decorate a Christmas tree in your home using AR. There was a visual cue to let the consumer know that the ad experience was augmented reality.
Unity is also bringing AR brand experiences to our supply of thousands of native apps powered by the Unity ads platform. This experience is using the private beta of our newest ads SDK with AR capability. Check out the amazing first campaign we are running with Disney for their DuckTales animated TV series. Our ads SDK delivers dynamic AR brand content into Unity-powered games; with the latest AR features we can create portals, product displays, animated characters and more, without the need to download an app; the experiences show up where the user’s attention is focused.
So this is where we’re at. We’re on the mountaintop — looking down at the land below. I’ve covered what I think is some of the best AR today, created by some of the world’s leading content creators and businesses. And the picture in terms of potential adoption is looking quite rosy. We’re approaching mainstream reach, and at the beginning of mainstream acceptance. The biggest players in the computer industry are continuing to make major investments in the infrastructure, the biggest brands on the planet are incorporating it, and, most importantly, customers seem to be loving it.
But we’ve got some near-term challenges, and some longer-term ones if we’re going to make AR stick for the long haul.
It’s All About Mobile (Mostly)
I know many of us are dreaming about AR using smart glasses — wearable, heads-up devices that are at least as powerful as phones, but light and comfortable on your face, for all-day use. We’re not quite there yet. Barring major breakthroughs in display technology — and possibly related innovations in edge computing and 5G networking — this is seeming like several years out. As far as headsets, which can provide a much richer experience and environment understanding than today’s phones, here’s what adoption may look like:
A Market study by IDC envision a healthy uptick in AR headsets over the next four years, but still looking in the small tens of millions… around where VR is today. One would assume that is primarily for use in the workplace.
Now, compare this to the projected growth of mobile AR handsets in the same period. We’re talking over 3 billion devices by 2021, according to ARTillry Intelligence.
For the next few years, we’re looking at a dual world of simple but ubiquitous AR on mobile handsets, and higher-end AR on headsets for industrial use. One the one hand, we have our familiar, everyday mobile devices now capable of delivering fun, informational and useful immersive content. On the other hand, we’ll be using headset AR to solve all kinds of problems in the workplace. Between the two, we’re all going to be exposed to 3D digital stuff floating in our view on a regular basis. But there’s no doubt, simply by the numbers, that mobile AR is how most consumers will be exposed to the technology. While it’s nowhere near as powerful as a Hololens or Magic Leap, phone-based AR is what is going to drive this industry forward for the next few years.
AR as a Feature
One thing that many of the phone- and tablet-based applications above share is that they have introduced AR into an existing solution as a supporting feature. Take the Yahoo! Mail app, for example. There have also been some great news and lifestyle pieces that use AR to enliven a story, such as a “virtual stagecoach” experience the ABC News App added during the Royal Wedding, or the 3D models of iconic David Bowie costumes that are available for viewing in AR within the New York Times app.
Getting back to games: Zynga’s wildly popular CSR Racing 2 introduced an AR mode allows you to display the cars you earned in the game in the real world, and take images or videos and show off to your friends. And Guns of Boom, Game Insight’s first-person shooter, does something truly novel with AR: they’ve created an e-sports spectator mode that gives fans the ability to watch their friends and clanmates do battle using a tabletop diorama of the game they’re watching!
This incremental approach is a great idea for designers that want to enhance something that already works by adding new capability. And it means that you can take advantage of AR without having to redesign your entire application; it’s not an all-or-nothing proposition.
I think most obviously, job one is the continued creation of great content and applications, with good design and utility.
To get there, we’re also going to need top-notch tooling. We’ve got startups like 8th Wall, and established players like Vuforia, supplying the underlying software and tools to create great AR across devices. And of course, my company, Unity. Let me spend just a minute or two on what’s happening with Unity. You’ve probably heard us talk about our three principles before: democratize development, solve hard problems, and enable success. We’re applying these to AR with as much vigor as we have for all our other platforms.
- Reach. Unity’s been there from the beginning supporting all the major AR systems. From integrating Vuforia early on, to providing first-class support for groundbreaking devices like Hololens and Magic Leap, to now fully supporting ARKit and ARCore with each new release as it comes out, Unity supports most platforms and, therefore, reaches the broadest range of devices.
- Multi-platform development. Unity is dedicated to its ‘build once, deploy everywhere’ promise, and has been the premier cross-platform development solution for mobile applications for nearly a decade. We’re extending this practice into AR by providing a single, unified API for both ARKit and ARCore, greatly lowering friction around developing and deploying. This is out now as an experimental package.
- Stability. Unity’s support for these devices and software APIs is first class, including ARKit, ARCore and Vuforia. This code has been built and tested by us, and is maintained and upgraded with each release.
- Fast iteration. We’ve been investing in features to speed up developers’ iteration time, including remoting tools. You maybe be familiar with Hololens remoting, where you can preview directly from within the editor while connected to the Hololens. We’ve developed a similar feature for Magic Leap, now in technical preview. And we’ve gone even further with Hololens remoting, with a feature called runtime remoting, which lets you run your application on a powerful PC in the cloud, and use the Hololens just as the display device and sensors. With this feature you can run way more complex simulations that would be possible with the Hololens, but get the untethered mobility and holographic rendering the devices offer.
More, better content on the largest number of platforms means more satisfied users, which means developers will be continue to be successful and continue this virtuous circle. Unity has been there from day one for all the AR platforms, and we’ll continue to be there. Also, not to be a tease, but… expect to see more AR authoring innovations from us in the very near future.
The AR Cloud
“The world is about to be painted with data”
Writer, Producer, Futurist, Forbes
We’ve got so many awesome speakers and visionaries at this conference. Many of the people featured in the AWE program are working toward a common vision, something that’s come to be known as the “AR Cloud.” It’s as much of a notion as it is a set of technologies. The concept is still taking shape, but in broad strokes, it’s the idea that the industry will converge on tools, technologies, standards and applications that put a layer of digital information on everything in the real world, accessible just by pointing your camera and expressing an intent. I think it’s best summed up by Charlie Fink’s pithy pronouncement: “the world is about to be painted with data.” It’s a big vision, one that is going take years to realize. If you can, make sure you catch one of the many sessions devoted to the topic.
Here is just a sampling of the technologies that could be involved in the creation of an AR Cloud. It’s a big list. The bulk of it involves the really hard stuff, like computer vision and machine learning. Other items are a little more straightforward, like content formats and search; they’re not rocket science, they just require a fresh look at old problems, and an open mind. Still another group of items is all about best practices and thoughtful governance.
- Image recognition
- Object recognition
- Face tracking
- Body tracking
- Background segmentation
- Shared State
- Instant, no-app access
- Universal browsing
- Content interoperability
- Search, discovery and sharing
- Privacy and Security
- Permissions and geofencing
I’m not going to go through all of them, but I’d like to touch on a few that I’m finding interesting right now.
Image and object recognition are gradually working their way into the phones via ARKit and ARCore. With these you’ll be able to recognize brand logos, furniture, and clothing, to name a few. Augmenting the body with face tracking, body tracking, background segmentation is showing up in dedicated apps like Snap and Facebook, driven by some amazing advances in computer vision, and will further blur the lines between the real and digital worlds by blending virtual content with people seamlessly. I’d sure like to see these features become more broadly supported in ARKit and ARCore, to make them accessible in even more applications.
Then there’s persistence and multiplayer. Here’s a quote from Matt Miesnieks, AR Cloud thought leader and founder of 6d.ai.
“If you assume that persistence and multiplayer are fundamental for AR, you need some sort of computer vision cloud infrastructure to support that.”
Persistence is the idea that our AR systems will be able to recognize the real world around us and then be able to build a virtual model of it that survives between application sessions. More than that, this virtual model can also be available between different applications; it’s a shared digital twin of the real world. Take it one step further and add multiplayer capability, and now that virtual model is not only persistent between sessions and shared among applications, but it’s viewable by multiple people at the same time, and looks the way it should based on where you’re viewing it. Think about it: an authoritative, shared model of the real world on which we can overlay digital information that we all share in real time. This is one of the holy grails of computing.
I think this is what’s getting a lot of people excited about persistence and multiplayer for the AR Cloud. However, at the risk of sounding heretical, I think their importance is being a bit overstated at this stage of the game. Bear with me.
There are plenty of great apps that don’t need these capabilities. First off: you don’t always want digital stuff to stay in your space permanently. For many use cases, it’s fine if that content goes away at the end of the session. But of course, that’s not true in other cases. If you’re redesigning your living room using AR, it would be really nice if that information persisted from session to session and you didn’t have to recreate it.
Second, synchronous multiplayer games are hard to design well for mobile, and with AR they’re going to be even harder. This is not to say there aren’t going to be great ones; I’m sure there will be. And looking beyond games to collaboration applications, I can definitely see a need for real-time, synchronous viewing of the same AR data in the same space.
Finally, not every AR experience is actually going to be based on a specific location. In fact, when we’re talking about mobile AR, you’re probably somewhere random in the house, at the office or on the move. Having a persistent model of the space around you isn’t necessary. And requiring someone to be in a specific place to play is damn inconvenient — unless of course the whole point of the game is to get you there, such as with Niantic’s location-based titles.
This is all by way of saying that I don’t think having persistence and multiplayer is necessarily a recipe for the killer app, nor is it currently a hurdle to adoption. So we shouldn’t let the lack of it hold us back when we design applications today. But, at the same time, we should pursue solving these hard problems with all vigor, because eventually, they’re going to be a big deal as the AR Cloud grows and the use cases mature.
“There’s vast potential for AR on the web — it could be used in shopping, education, entertainment, and more.”
JOSH CARPENTER, Google
Last but not least, I’d like to talk about the web. You may have heard the terms WebVR, WebAR or the umbrella term WebXR. Over the last few years, the people who make browser have been integrating with various devices like the Rift and the Vive, allowing web developers to create VR applications that don’t require a download or install. This opens up a lot of possibilities for immersive content that wouldn’t be otherwise possible: share a VR experience by sending a link in email, or in a tweet; the recipient can immediately enter and access it. For short-form content like casual games and e-commerce websites, this is the way the web works, and it would be great to have that same ability with XR. Now, the browsers are experimenting with expanding the scope of the browser to incorporate AR, initially for mobile ARKit and ARCore. This is possibly an even more enticing proposition, because it can reach those billion AR phone users with a natural extension to today’s browser functionality. Done right, this is going to enable even more content development, faster, without friction, and reaching even more people.
In addition to browser tech, we need an interoperable format for delivering the content and sharing it between applications. If you haven’t, look into glTF, the transmission format standard for 3D assets. It’s essentially the JPEG of 3D, supporting full 3D objects, scenes, textures and materials, animations and skins, and more. glTF is maturing nicely, in its second version, and is supported by all the major tech companies in the space. Facebook uses it for its 3D Posts, Microsoft has built support into its first-party applications like Office, and even exports glTF files from Minecraft for sharing with the world.
Bringing these things together: the AR Cloud will surely partly be stored as glTF assets, and accessed by universal client software such as AR browsers. Add image- and object recognition capabilities and you can start to imagine a future where the real world is alive with data, accessible at the touch of a button in the blink of an eye. Now THAT is an exciting vision of the future.
But first things first. 2018 is the year of deployment. The stage is set for AR to be an everyday thing. Let’s make this immersive revolution stick with killer apps, real business solutions and meaningful experiences that are achievable today. It’s in our grasp.