What AR, VR and AI mean for storytellers

How visionary creatives mash up tools and frameworks to tell new tales

“Any sufficiently advanced technology is indistinguishable from magic.”

Arthur C. Clarke

Humans have an innate desire to share stories. We’ve been doing it for thousands of years, from tales told around the campfire, through legends written in hieroglyphics, to the digital content we publish and consume online.

While our inherent desire to share stories remains constant, the technologies continue to change. Advances in Augmented Reality, Virtual Reality and applied Artificial Intelligence present new opportunities for storytelling. We’re seeing early experiments as visionary creatives and technologists mash up tools and frameworks — and the pace is only going to increase from here.

How can storytellers and content creators keep abreast of this rapidly evolving space? Let’s look at some of the fundamentals of storytelling in context, an overview of the technologies and some early applications to get up to speed.

New technologies, new experiences

Just as weblogs helped create the blogosphere, and video platforms paved the way for vloggers, casters and streamers, new technologies are building foundations for new types of storytelling. In this post, I’m looking at three specific technologies.

• Virtual Reality (VR): Where the technology allows the user to feel as if they are immersed in another world, with sensory inputs (audio, visual, haptic) reinforcing this belief. This technology gives the sensation of presence. It makes you feel like you’re there.
• Augmented Reality (AR): Where the technology adds a layer of experience to the real world. In some ways, this is the inverse of VR. With regards to the experience, this technology makes it feel like it’s here.
• Artificial Intelligence (AI): AI is a blanket term for a wide range of applications. In the context of storytelling, AI is helping to make AR/VR experiences more memorable. Particularly relevant technologies are image recognition, video manipulation, voice recognition and conversational interaction.

Pikachu, I choose you!

A milestone moment was the release of Pokémon Go, which exploded into popularity in summer 2016. Sure, it helped to have a massively popular franchise, but it was the game’s innovative use of mobile technologies that brought imagination to life. People could realise their childhood dreams of finding Pokémon in their parks and neighbourhoods. The game generated $600M in three months and proved that an Augmented Reality service could have mainstream appeal.

Evan Spiegel, CEO of Snapchat claimed that their dancing hot dog was the world’s first augmented reality star, with the filter having been viewed over 1.5 billion times. More recently, Snapchat partnered with artist Jeff Koons to display virtual sculptures around the world. Apple, Microsoft and Google are building tools to help developers create new experiences, like Ikea Place, an app that lets you unlock your inner designer, placing virtual furniture into any real-world space.

These examples are interesting, because the phone’s camera isn’t being used to record a photo or video for later consumption; but instead, having us use the camera in real-time, to see experiences taking place in the world around us. Under the hood, advanced AI algorithms are helping to make these experiences more lifelike, smoother and more efficient to run on devices.

We’re also seeing AI help power other aspects, including virtual characters. While text-based games have been around since the 1970s, recent years have seen a resurgence, with games and experiences where users can ‘chat’ with characters in real-time.

Massively, a Toronto based company, has been building virtual characters based on characters in movies, working with Paramount to create chatbots for the Teenage Mutant Ninja Turtles. The company is now using voice technology to power character bots that you can physically talk to, over platforms like Amazon’s Alexa.

“Storytelling is evolving, and new technologies are helping to build deeper, more lifelike experiences. It’s now possible to be part of a story experience, and have a real conversation with your favorite character.” Russell Ward, President, Massively.

Advances in cameras, sensors and Internet of Things technology are helping drive this. We already have cameras in the ubiquitous mobile phone, but we’re beginning to see headset technologies taking off. Snapchat’s Spectacles (Consumer-centric) and Microsoft Hololens (Business-centric) play in the AR space, while in VR, competitors are working to seize new markets, with Occulus Rift, Samsung Gear and HTC Vive focusing on gaming applications. Platforms like Microsoft’s AltspaceVR and Facebook Spaces are creating opportunities for developers to test out new ideas.

“…We think about how can we create experiences that are useful for people and that resonate with people. That’s the phase we’re in right now, versus expecting 2 billion people to necessarily run out and be in VR right now. That learning phase is important to find the magic utility that will make people aspire to be in VR … really I believe it’s a way to connect people when they can’t be together. That’s meaningful.” Rachel Franklin, Head of Social VR.

Narrative techniques in AR and VR

How do you lead a user through an open world? How do you provide guardrails without restricting the experience? As new media emerge, creators must work out — through trial and error — how to deliver the best experience. Techniques, frameworks and best practices were developed over time for radio, film, television and games, and creators are now defining these for AR and VR.

Storytellers in these mediums should see themselves as guides, not directors. When designing experiences, storytellers must consider three unique attributes:

• Presence: The sensation that you are a participant in the augmented or virtual environment. This includes visual, audio and even touch inputs. There’s no fourth wall here; you’re an active participant in the environment. In an Augmented context, elements of safety must be considered — we don’t want people falling off cliffs chasing Charmanders!.
• Interactivity: You can interact with the environment around you, and it, perhaps can interact with you. In a game context, you could walk through a forest, hunting virtual creatures. If you find a shack in the woods, you can open the door and step inside. The experience must be responsive to the player, but storytellers must also consider how players might interact in multi-user environments.
• Influence: Open environments provide great opportunities for discovery, and users can craft their own stories. Storytellers need to use prompts, queues and tips to help guide a user through an experience. A great analogy is an NPC in an open-world video game like Skryim, who might give you quests to complete or offer hints of what you should do next.

Stories from another world

How will these innovative technologies and growing platforms provide new canvases for storytelling? I’m excited about a number of categories and have shared some examples here:

• 🎮Gaming: Imax VR arcades drop adrenaline junkies inside blockbuster video games.
• 🏈Sports: Microsoft Hololens brings the thrill of the game to your living room with insights, social interactions and players in your living room.
• 🎸Concerts and events: NextVR teams up with Live Nation to help fans rock out to concerts through virtual reality.
• 📰Journalism: New opportunities (and challenges) in how we report on, and understand news in our world.
• 🎂Experiences: Friends and family can join you across space (and time) to share moments together, from birthdays to weddings to just hanging out.

There are many applications, from recruitment to education, medical applications and more. This month, Microsoft announced their Mixed Reality Platform with visors from multiple companies. Studies suggest that the global AR/VR market will exceed $108 billion by 2021. In the coming years, we’ll see advances in the capabilities and ubiquity of AR/VR and AI powered technology, increasing appetite for new experiences and pioneering examples of creativity.

Ready Player One

It’s an exciting time ahead for storytellers, influencers, marketers, content creators and entrepreneurs. While we’re still in the stories around the campfire stage of these new technologies, there are some actions that you can take to prepare yourself.

1. Learn about AI, AR and VR technologies: It’s valuable to keep up to date with how these technologies are evolving. It’s not necessary to become an expert in these domains, but it’s worth learning the fundamentals.

2. Try out AI, AR and VR experiences: Filmmakers, writers, game developers, marketers and agencies are creating new experiences all the time. It’s worth trying out those that you can, so you can keep abreast of how the experiences are evolving, what works and what does not.

3. Think of your story as an experience: Rather than a 30 second ad, a blog post or an advertisement, how will your story work in as an interactive experience? If a user can talk to, touch, feel your story… how should it respond?

Are you using AR, VR or AI to power a storytelling experience? I’ve love to hear your comments on what’s worked, what hasn’t worked and what you’re most excited by.