Holo-Moments — “Sanic Hegehog (sic) and Its Implications for the Future of Mixed-Reality Storytelling”

photography by Jennifer Lopez

My name is Matt Bretz. This is my second blog post in our series on HoloLens development. I’m a creative director with a background in writing and directing, finding my way with the team through creating mixed-reality (MR) experiences.

At Ayzenberg, we are first and foremost storytellers. And yet as we’ve embarked on developing mixed-reality apps for the Microsoft HoloLens we aren’t obviously tapping into that expertise. We’re spending most of our time on an application for building code inspectors. We’re helping these folks work more safely and efficiently as they endeavor to keep us all safe by ensuring the structures we work, play and live in, meet agreed upon standards for health and safety. And of course, it’s important to bring a sense of storytelling to software engineering. But I don’t think anyone who knows us well would look at where we are focused now and not kinda go — “Huh, Ayzenberg made this?”

Truth is, there a couple of important reasons that we’re focused on a building code inspection app. One is that we’re following the lead of the folks who know HoloLens best — Microsoft. And as they are principally software developers, they are rightfully careful about getting pigeon-holed as an entertainment device. They’d much rather be appreciated as the future of screenless personal computing. So most of us close to the mothership have made our focus developing applications that transform the world with holograms at an enterprise scale. But the second reason we at Ayzenberg haven’t yet played to our strong suit with the HoloLens in developing a killer story based experience is … it’s really friggin’ hard!

Here’s why: Great stories are woven from five threads — who, what, where, when and why. Master storytellers control all of these elements and are doing something cool and unique in each of them. But when you pick up a HoloLens to tell a story, you have given up control of four of these five — straight out of the gate! You can’t control WHO your protagonist is. It’s whomever picks up the device and puts it on. You can’t control WHAT they will do because … well … they have free will. And because a fair quotient of the real world is visible in any mixed reality experience, you can’t control WHEN your story is set — it is hard to escape “now.” But perhaps most significantly, as compared to most video games and VR experiences that similarly relinquish control of some of these story elements — in the HoloLens you cannot control WHERE your story takes place.

Inside — Playdead

But despair not — humans are always figuring out new ways to tell stories in new mediums! So brilliant mixed reality entertainment experiences are doubtless on the way from Ayzenberg and many others. In the meantime, the closest we’ve come so far is a little mini-game we call Sanic Hegehog. Without going too far down the wormhole, Sanic is the purposely lo-fi, alter-ego of Sonic the Hedgehog that Ayzenberg created for Sega’s Sonic social media channels. We invented Sanic so we’d have a character to use in content that didn’t require lengthy brand approvals. And recently, we created a HoloLens mixed-reality game in which we distribute Sanic’s rings throughout the real world of any room in which a player finds themself. The player then platforms Sanic through the real world to collect the rings. Having clearly established with the creation of this Sanic Hegehog experience that Ayzenberg is at the front edge of making entertainment in mixed-reality (spit take here please), we’ve got a few insights to offer already that we hope might help others leapfrog with us down the path to success in mixed-reality storytelling.

To start with, one of the ways to look at the challenge that storytelling in mixed reality presents, is to look back at a brief history of storytelling in a number of different forms. Each form presents an innovative advantage over those that came before. And each comes with some limitations. By focusing on why we might give up some “who, when and where” — and in exchange for “what” — we hope to identify some promising principles for storytelling in HoloLens.

So in the very beginning … presumably there was a time when things happened, but there was no one around to tell stories about them. Still … one way to look at the way things happen in the world is … 1) there is a way things are … 2) an event occurs that affects those things … and 3) a new way things are emerges. A beginning. A middle. And an end. And it could be just a coincidence, but that linear, three-piece unit — a beginning, a middle and an end — became the foundation on which most stories of olde were built. And in ye olden days, these stories were told verbally. Bards like Homer in ancient Greece roamed the land weaving tales of mankind and gods that helped explain the human condition. What was special about their ability as great storytellers was how they “stacked” these three-piece blocks into longer and bigger creations — like the Odyssey — that nevertheless recapitulated the pattern of beginning/middle/end upon which each individual story beat was built.

We haven’t exactly achieved the same level of storytelling as the Odyssey with Sanic Hegehog yet … but more on that later. And of course, there are plenty of folks who will take issue with this slavishly linear analysis of storytelling. Indeed, there are other ways to structure stories. But I’ve chosen to focus on a linear progression from a beginning through a middle to an end here because it is arguably the most challenging structure to accomplish in the HoloLens, while at the same time the most iconic form of storytelling.

But back to Homer … great oral storytellers were famous for their epic yarns because they were exciting! People would come from all over and stay for long hours to hear them because they wanted to know what happened next. And even when one tale ended, they were hopeful that another would begin. It’s worth noting that telling oral stories isn’t terribly conducive to audience participation. So if any of Homer’s audience were craving interactivity — they had to turn to other outlets like the church or the Coliseum (oh, wait … that was Rome) But the real shortcoming of telling stories aloud was “reach.” Homer could only go so many places and see so many people. And as others tried to learn his stories and take them out into the world, there would inevitably be a fair amount of telephone tag bastardization of the original words, which for an auteur like Homer, had to be a little galling.

So writing came along. Writing stories down didn’t make them any more interactive — in fact it probably made them less so. But it did make it easier to share them. They still tended to be linear. And “agency” in written storytelling was principally concentrated in the writer — meaning the writer controlled the who, what, where, when and why. But written stories may have given a taste of agency to the audience because they provoked … the imagination. Like it or not, the writer has to accept that the readers’ interpretations of the written word are a part of the experience of reading a book. No matter what they write, it will take each member of their audience to a somewhat different place.

Call me crazy, but I have to think that part of what drove storytellers to more visual mediums like theater and film was the opportunity to leave less to the imagination. In a movie, you can tell your story in a way that looks and sounds pretty much exactly as you envision it. Still not a lot of obvious interactivity with the audience (unless it’s The Rocky Horror Picture Show). But I think that these visual mediums of storytelling also created a bit of a monster in their audiences. Because experiencing a story in pictures and sound is so much like life … we began to wonder “why couldn’t it just be life?”

Why couldn’t we — the viewer — be given agency in the creator’s world? So WE are the WHO enjoying the writers’ WHAT, WHERE, WHEN and WHY. This is where games as a storytelling medium re-enter the fray. I say “re-enter” because games have always been a beginning-middle-end based form of entertainment. While stories build beats of beginning-middle-end on top of one another, a great game may essentially repeat the same pattern over and over again — scratching the itch we have to do it better and better. With Sanic, for example … I see the ring. I reach the ring. I win. Whether it’s me in the first person in a game like basketball. Or me in the third person controlling a proxy like Sanic, in a game … I have agency! And as the fidelity of video games neared cinematic production value, storytellers flocked into the video game space to give their audience the agency they craved.

Now, for the sake of time, I’m going to point out that the challenges of telling a story in VR are just a short hop away from those posed by AAA, first person video game development. So let’s make the leap to talking about that. In exchange for player agency, in VR we have given up control of what the hero thinks, where she looks, what she does, in what order, etc. So how the hell are we going to tell a story, right? Well, game developers have paved some proven roads in this area. You create levels. So one set of beginning-middle-end triggers another triggers another. Players have agency. But they can only go certain places at certain times. They all have to get certain stuff done and visit certain places before they can move on. Not everyone experiences everything in the same way. But they all get where you want them to go.

One of the key tools for these kind of “cheats” in storytelling without control of the WHO (the hero) is — the environment. The WHERE. For one thing, when you control WHERE you can usually communicate WHEN as well. And if you are crafty you can hide “rails” in the environment that guide your players where you want them to go. You can put “user experience instructions” (UX) wherever you want. UX is also a great way to communicate WHAT and WHY. As long as you have WHERE. Which in VR, you very much do.

Which finally brings us back to the essential challenge of telling stories in mixed reality — we’ve given up the WHERE! And everything that comes with it! How on earth are we going to create beginning and a middle and an end?!? It’s time to seriously consider Ayzenberg’s most marvelous mixed reality creation to date … our Odyssey, if you will … Sanic Hegehog. What is so friggin’ great about it?

First, it’s actually more of a game than a story right now. And herein lies what we’ve found to be a critical lesson in mixed reality storytelling. Start small. Then build big. Sanic is working the single building block of beginning-middle-end that most games do. All we want to do is get better at running him around and collecting rings. The same story over and over for now. But now that we’ve got that up and running, we can iterate on the next two building blocks — after we 1) collect rings … 2) Eggman attempts to destroy us. And …3) we defeat him! What we are doing in level one mixed reality with one unit of beginning-middle-end teaches us what we need to do next. Don’t try to eat the whole elephant. Eat it one story-driven bite at a time.

The second takeaway we’d offer from our Sanic experience is — consider third person. To be in mixed reality, you have given up the WHERE and the WHEN — think about starting with an experience in which you preserve a bit of WHO by inventing a hero that you guide through your WHAT and WHY. No question that down the line, MR developers will be making first-person masterpieces in mixed reality. But there’s no reason you need to start there.

Finally, I’d offer that Sanic has taught us … when mixed reality gives you no WHERE … make NO-WHERE-ADE! Turn the unlimited and always changing physical reality that is your canvas in mixed reality to your advantage. That is truthfully the only cool thing our Sanic app does right now and we kind of happened on it by accident (longer story for another time). But the real insight here is — make the environment a character. Part of your WHAT and WHY. Remember that if you were in VR you would have to build all that stuff in 360-degree interactive glory. Whew … you dodged that bullet.

The bottom line is that as in any medium, when you develop stories for mixed reality you make certain sacrifices. So your starting point in ideation needs to be the advantages for which you made those sacrifices — or else … why not make VR, or a movie, or write a book? In MR that means starting from how can you use any environment in which your audience dons their HoloLens as a principle character in the narrative? How can you celebrate your audience’s heads-up and hands-free agency in the story? How can you start today with one little byte of beginning, middle and end — so that by the end of the year you are MR’s Homer? Or, at least, Sanic?

Ready player you.

photography by Jennifer Lopez
One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.