Best practices for mixed reality experience design in an open world
All immersive experiences are contained — either by hardware (a closed-off VR headset), a pre-set space (a black box theater, gallery, or building), or the art itself (single AR objects tied to one moment or marker). What I have observed is that this seems to be the preferred way to create immersive experiences and AR… I mean… it makes sense, right? The more control you have, the less to worry about when trying to pull off an already complicated consumer, audience, or user experience. But what about the future of immersive digital entertainment? With the promise of a consumer-attainable AR headset in the near future (who are you betting on?), mixed reality storytelling is going to take to the streets in a big way. Creators, with their infinite imaginations, will need best practices in experience design for uncontrolled and unpredictable environments. Similar to the birth of the touch screen, UX will be upended by spatial computing where the real world presents designers with new challenges.
Over the last four years, I’ve been solely focused on creating mixed reality experiences within these uncontrolled environments on the streets and landmarks of New York City. These are not just small contained demos but full-feature experiences lasting over an hour. These experiences are made for the real world we walk on every day that is filled with people, noise, traffic, and tries to consider the unpredictable — like a random helicopter landing nearby or a vendor that decided to show up today.
Given there is little written on the experience design for this new type of storytelling, I thought I would share what I have learned through making linear and non-linear mixed reality experiences that are interwoven into Times Square NYC and one of the busiest national landmarks, the Brooklyn Bridge. All these learnings culminated into my app, ChalkNotes, and experiences of NYCXR. ChalkNotes fills a hole in the mixed reality landscape — a single no-code app to create and distribute mixed reality experiences to your community.
Here is a brief summary of learnings I have observed that required tweaks to design along the way. Even considering all of this, there will always be a “black swan” that drops a bomb in your experience — be ready for those.
1 — People. Perhaps the most obvious to start with is people. The real world is full of them, and they can appear at any time. People traffic can be somewhat predictable but fluctuates in surprising ways. You have to spend extra time scouting a location. (A) Upfront, address “the people problem” with audiences through a preamble. I use a starter audio track to provide expectations and framing of what audiences will experience on their journey. This is mandatory not only to set rules of engagement but also for safety. (B) Don’t ignore people. Accept the reality that people are present, mention them and the volume in the story; they are players in your experience, but also not in the know — you can use that to your advantage. (C) People are the most unpredictable; one morning a person will decide today is the one day they will put their cart in your space, close their shop, or play music real loud.
2 — Noise. Another obstacle that is everywhere. (A) Visit your experience locations at various times of day. (B) Expect the unexpected — music, car traffic, people talking, air traffic. These will get in the way of your subtle sound design, so don’t be too subtle. (C)Do not assume everyone owns $200+ earphones. Sound mixing levels will have to consider the average decibel of a user’s surroundings.
3 — Time. More specifically, the time of day and how much time someone has in their day. With the future of mixed reality, you can contain experiences with scheduled times, but to maximize the commercial opportunity, you should let mixed reality experiences run 24/7 for massive scale. (A) People need the flexibility to do experiences on their schedule and not one dictated for them. So consider what time of day your mixed reality experience is going to be consumed and how long the experience will last. (B) UI (user interface) colors need to adjust for apps and glasses depending on what light is coming from the sun to reduce any strain in reading your app. Dark mode is your friend. (C)People traffic varies over different times; if you need fewer people, schedule your experience during off-peak hours. (D) There has been research showing no single stop should have an experience that lasts 2–3 minutes, but I have seen that if the experience is engaging, participants will spend 20 minutes at one location. (E) Honor people’s time and do not be repetitive. Incorporate surprises within the experience to keep momentum forward.
4 — Weather. Weather can make or break your experience. It is a big player in what an experience can be. (A) Tell audiences to enjoy the possibility that weather may be a part of the experience. Get ahead of it before people decide that sunny is the only time to enjoy. Weather, like the time of day, affects the user’s experience. (B) Make weather part of the experience with a weather tracker in your app — change the experience depending on temperature and/or precipitation type.
5 — Hardware. IMO, we are still in a pre-AR glasses world. What is offered has too limited a field of view and is expensive. Because of this, most rely on their phone’s screen to enjoy mixed reality. Until these come about, we are stuck with hand-held devices. (A) The participant’s device should feel like part of the experience, not separate from it. (B) Devices should guide an experience and not be the experience — the goal is to get the user to look up, not down at their phone. (C) Let’s talk battery life — can someone already make a better battery already?! Be aware of the battery limitations of several different devices and minimize rendering so the device does not get too hot. (D) It’s obvious to most UX pros, but you should test experiences on various different devices — hardware limitations drastically change the experience from person to person.
6 — Affect the space. Many stories already exist about the world around us. Here are some elements I find work. Both your story and your AR objects need to feel part of the surroundings and part of the story. Think of the world as a movie set that the audience is wandering through. To keep the story connected to the space, you cannot ignore the world around the audience — the world must be integral to the story. (B) Use AR objects to make your “metaverse” feel lived in. Pretend characters have been to a location by leaving an artifact. (C)Make the audience part of the story. Suspension of disbelief can be powerful, but don’t rely on it the whole time.
7- Physical locations. Linked back to people who control our open spaces, physical locations change as well. Places have office hours, suddenly go out of business, construction happens… the world grows. (A) Have your story only rely on solid nearly unmovable things. Don’t have audiences use movable things like a table but concrete features on buildings. Unless that is the intention, then you should set the schedule to match when the referenced real-world objects will be removed. (B) Unless you have permission, do not alter the physical space — try to use what is there and let AR objects and audio carry the story.
8 — Navigation. (Direction/Instruction). There are two aspects to navigation — direction and instruction. Today, most people use their mobile phones/tablets to experience mixed reality, equipped with tools to determine the relative position between a person and a destination. (A) Avoid assuming that audiences know where they are going. Provide in-app or third-party walking/driving directions, a compass, distance to the location, and, if audio is involved, discuss the listener’s surroundings (e.g., instruct them to look up in the script). (B) Be very specific in audio and helper text if you need someone in a particular location. In a park with hundreds of trees, find unique markers to direct your audience toward.
9 — Audience decisions. Don’t Make Me Think is a great UX book and mantra that continues to inspire my work (also Nielsen & Norman’s “You are not the user”…look up false-consensus effect) Ensure that it is very obvious where audiences should and should not go. Restrict audiences from venturing elsewhere. You have to keep the experience tight; otherwise, your audience will become disengaged and wander off the path. (A) Determine if you want your experience to be more open and flexible or intense and completed in a timely manner. (B) To create a truly flexible experience, build choice into your app experience. ChalkNotes provides a choose-your-own-adventure-like tool where the branches of a story can expand and contract infinitely — putting the audience in the game.
10 — Social. AR/VR/MR is generally experienced alone. Even in live theater, if you go with a guest, you still sit in the dark and wait for intermission to have a conversation. However, do not be afraid of creating an experience that is generally solo. Create other ways for audiences to socialize. (A) Consider where you can insert social moments into your story. Your distracted audience will be plugged in with earbuds and eyes on the world most of the time. You will have to remind them to be social. Build in moments within the experience, either in the script, app reminders, or leave space between locations in an experience for time to be social. Actually provide instructions to stop for a moment. (B) Another option is to provide passive socialization even if participants are apart, or if on-site, let users leave something digital behind. I saw an excellent example of icons representing people watching AR theater remotely. ChalkNotes uses a shared chat channel per experience, and JR’s SuperBlue experience lets users leave images and sound at pre-chosen locations.
These rules should continue to evolve as the demand for storytellers becomes increasingly challenging to fulfill. I look forward to seeing what you create out there.
>Phil
