Envisioning Holographic Experiences
A Designer’s guide to Mixed Reality
Designing for Mixed Reality (digital “holograms” blended into the physical world) is a new frontier for everyone, and definitely requires some experimentation to get right.
Envisioning helps define the journey and speed the process.
Summary
Holographic computing (literally) demands a new design approach.
I’ll cover how you can quickly conceptualize and clearly communicate your early ideas for holographic experiences without using expensive software or needing a Game Designer’s skillset. You’ll learn some handy shortcuts to create, test, and refine your overall holographic experience (with or without coding) to move your ideas forward.
SPOILER ALERT: Understanding the people part of the holographic equation is the key to envisioning truly engaging holographic experiences —
it’s never about the technology.
Part 1 — Why design for Holographic?
You can count on one hand the number of truly breakthrough computing platforms that have come along in the last few decades — Desktop, Web, Mobile, Cloud. Holographic is now without question one of them.
And not surprisingly, holographic is perhaps the most challenging platforms(from a design perspective) for early adopters to take full advantage of due to its fantastical nature. It is exactly what we’ve seen in our favorite movies and read about in sci-fi novels for so many decades. Perhaps even better.
Seeing and interacting with a hologram for the first time is truly memorable. For some, even profound. It excites the mind and stokes the imagination with blinding speed. We imagine possibilities and new realities in an instant. But, don’t take my word for it. Try it out for yourself to see how transformational really is.
All that said, we are still in the very early days of holographic computing. Everything is possible. Nothing is guaranteed. Perfect time for Designers to explore the outer edges of the frontier in search of truly breakthrough experiences.
New Frontier, New Approach
Given the availability of devices and free toolsets for Windows Holographic, now is a great time for anyone to jump right in and start experimenting with Mixed Reality (the term most frequently associated with holographic computing).
Just one catch thought — designing for Mixed Reality requires a completely different mindset, language, and approach than just about every other major computing shift that has come before it. You won’t nail your first holographic project by relying on your flat screen design experience as a guide. Design fundamentals aside, there are some unique aspects to master.
Turns out holograms need you to grow as a Designer.
Mindset — you do need to think differently than you’re used to, but not for the obvious reason (3D). It’s the people part that needs deeper exploration. Turns out you are the interface. You need to focus on how people respond to their new environment and input model rather than the mechanics of a standard interface. As a Designer, you’ll find yourself mentally switching between thoughts of the participant, the role of an observer, and qualities needed by holograms quite often. That’s quite a mindshift. It requires imagination and empathy to put yourself inside other people’s thoughts, motivations, and expectations all the time.
Language — to express yourself clearly, you’ll need to get comfortable thinking and acting in three-dimensional space. In that regard, becoming fluent in 3D is just like learning a new spoken language — it has it’s own unique flow, idiosyncrasies, and accents. The only problem is that you can’t go out and buy Conversational Mixed Reality to learn how get in synch with your audience. It takes practice, but you’ll get there.
Approach — the truth is very few of us have mastered using the room itself as the interface and ourselves as the controller. Almost everything before this has been all about controlling what’s behind the glass — now we find ourselves sharing physical space with crazy digital objects. You have to mentally pivot to design completely outside the glass and integrated into your physical world. It’s fun, but a challenge at first.
Holograms should not be flat
If you are thinking about porting an existing piece of software or content that’s inherently 2D and flat into the holographic world, please don’t. The world of Mixed Reality is a new frontier that deserves your best thinking. Use the opportunity to reimagine what that idea could be like if it was brought to life as a fully realistic, multi-dimensional entity, that felt as real or as imaginary as you can make it. We already have Augmented Reality (AR), let’s not repeat ourselves inside MR. I know it makes sense to have callouts and so forth in games and certain types of content. But, now’s the time to leap ahead, not fall back on old paradigms.
Ready?
All that said, this is not rocket science or brain surgery. Designing for Mixed Reality is all about understanding how to engage and excite people. And frankly that’s one of the most fun things you can do.
Let’s get started exploring this new design space by envisioning the holographic experiences that you’d love to bring to life. It would help to learn some of the basics of Windows Holographic to ground your work a bit, but let’s jump right in and talk about how to make some quick progress with your breakthrough experience ideas.
You’ve got this.
Part 2 — Why Use Envisioning?
Being able to mentally picture then tangibly communicate your thoughts on holographic experiences is a pretty fundamental skill in this new space (given the amount effort involved that follows).
And that’s precisely what envisioning is for.
Envisioning helps people understand what’s possible or potential in a new realm by using whatever means necessary to depict how it would feel.
Envisioning can be done with words, pictures, actions, storytelling, animation, or any other way of communicating. The important part is to get your ideas across quickly and clearly. In fact, envisioning at its best is often more like a quick sketch than a polished cinematic masterpiece.
Sometimes, what is depicted in your explorations doesn’t even have to be real to accomplish its goal.
Realistic vs. Futuristic
It’s important to note that by its nature envisioning does not have to be completely realistic to the current capabilities of the platforms or tools that currently exist. Envisioning is generally “of the future” and as such can feel more inspirational than factual.
As Designers, part of our job is to push the boundaries of what could be to illustrate how things should be. Conceptual design or envisioning is a great way to do that. It poses hard questions, raises concerns, pushes engineering to consider, and creates dialog to further explore the issues. At its best, this type of envisioning is inspirational and provides a North Star for aspirations and dreams.
For example, in the HoloScenes series there’s an underlying premise that anyone can see the holograms in open spaces because they have been publicly shared and the technology used to see them is prevalent and frequently used. None of that is true today, but that’s not the point. It illustrates a possible future that we can work toward.
Envisioning can also be used to show your ideas quite realistically, remaining true to existing capabilities and norms. This type of design is done to help people visualize something that has yet to be built. It’s meant as a blueprint of sorts rather than a directional suggestion. The real difference between this type of envisioning and a more fanciful treatment of the subject matter is that people can recognize the realistic nature.
Combining both realistic and futuristic aspects of ideas is my favorite way to work in this medium, as you’ll see throughout my examples. You can combine established practices and capabilities with hopeful improvements to take your audience where you need the story to go — whether to push on the technology aspects of the project and platform, or to set new ideas forth for interaction and participation possibilities.
Envisioning is what you need it to be, realistic or fanciful.
Capturing ideas
In the case of being more realistic with your explorations, you don’t need to perfectly visualize and exhaustively define things upfront in holographic to successfully communicate the concepts — just the opposite in fact. Since speed is the key to moving your idea forward, doing a hand drawn sketch or crafting a narrative about the feel of the experience you’re looking for is a great way to start the envisioning process.
You’ll have many opportunities to refine and fill out the details of initial explorations, so don’t slow down in when you need to capture that brilliant thought about where things need to go.
Conversely, if you have the skills to jump in and do your initial sketches in code, 3D models, or high fidelity mockups go for it! That runs counter to traditional Design methodologies of staying low fidelity at first, but there’s nothing traditional about Holographic.
There’s no “right way” to envision your ideas.
Just do it fast and keep moving.
Inspiration is everywhere
The envisioning process starts with consciously observing the world around you. It gives you a baseline. Sounds obvious, but trust me, it’s not that simple. It takes practice to be very observant and conscious of how things appear and behave in the real world. There are so many subtleties, so much incredible detail.
Any experienced Designer will tell you the same thing when asked about how they get better at their craft — intentionally seeing the richness of the real world helps train the eye and the mind. That ability to recall or picture how things look and behave later on when they’re not right in front of you is something to master.
The ideas for many of the best holographic experiences come from ordinary objects and situations in the real world, not the digital realm. Kinda weird to think of it that way, but it’s true.
“Look closely at the world around you, then let your imagination to show you what could be.” — fake fortune
Try drawing some inspiration from the real world each day, both natural and technological. Pay attention to how things work. The physics, the light, mechanics. Lean toward the things that move you. Use your gut instinct to set your course, not someone else’s data about what verticals are going to be hot for Mixed Reality.
Did you see that?
Training your mind to look deeply into the real world is only half of the envisioning equation though — the other is the ability to visualize on demand. That’s not like daydreaming. It’s a purposeful, directed conjuring of imaginary things.
People will think you’re weird for staring blankly into space, but whatever. Your imaginary holograms will be just as you want them to be, and perhaps even more intriguing than we’ll be able to produce. Never mind that. Conjure at will. You’ll find that once you start exercising that muscle it gets to be second nature and incredibly fun.
Any Mixed Reality Designer will tell you that being able to imagine and visualize things that other people don’t see is one of their most important skills.
Can you imagine the emergency light fixture above jumping down and interacting with you as a charming chameleon robot? OK, didn’t think so. But, I did. And it was fun.
Wherever you happen to be throughout the day, take the opportunity to imagine what’s missing from the space you’re in or what would be a really pleasant surprise to encounter there. You’d be surprised at where your imagination takes you without trying very hard.
Take this scene below. What do you imagine behind the glass?
Correct! There’s a T-Rex as big as a house right on the other side of the MoMA lobby glass wall. Oh. Wait. I guess I forgot to draw that in above. My bad.
You get the idea. Holograms can be anywhere.
WARNING: visualizing holograms is addictive
For me, seeing holograms where they don’t exist is something I can’t turn off that easily. My mind naturally wants to fill empty space with holograms. It’s so easy to do and often happens without consciously thinking about it. Occupational hazard I suppose.
For example, I walked into a hotel lobby one night and imagined this…
Sure, making whatever you desire appear on demand is super cool. But, there is a bit more to this space filling than merely conjuring at will. You want to be purposeful, so of course there’s a few good ways to do that. Let’s talk about two different aspects of adding holograms to a space.
To augment or add?
Remember that holograms can be used to both augment existing things or add new elements to any space. It’s a subtle distinction, but worth talking about quickly.
Augmenting existing things with holograms should enhance them. It’s a great effect to see something floating on or near an “ordinary” thing, but not all that shocking given the number of times this is depicted in sci-fi movies. In the real world you’ll see good examples of this method in many Augmented Reality (AR) systems, where they typically layer additional information on top of the real world to help you have a better understanding of something.
Adding new elements to a space using holograms will generally be more of a surprise to people, which is a great opportunity to blow people’s minds. They probably weren’t expecting something appear in that way, so it’s all new to them. The trick is mixing holograms into the real world in such a way that truly feels integrated and right. Even with great tools and some experience, that’s easier said than done.
We’ll end up blending both of those techniques into a seamless experience fairly often, and in fact that’s a hallmark of some of the best holographic experiences — they bring something new, but behave in an expected and seemingly natural way.
Part 3 — Envisioning Techniques
The goal of envisioning holographic experiences is to capture the essence of what we have imagined, then clearly and persuasively communicate it in a form that anyone will “get” without effort. How we do that is surprisingly easy sometimes.
The method we chose to envision is sometimes the key to achieving our goal.
Something that looks like a sketch (hand drawn, lower quality, hastily done) sets an expectation that it’s still in the idea phase. It’s a suggestion of what is to come later during refinement. Using lo-fi sketching is generally quicker to get done, and can lead to generating more ideas.
Using high fidelity techniques are effective in portraying a more realistic scene. They can take you right there and have people understand exactly what you are after through photo-realistic rendering and real world photography combined with your realistic visualizations of holograms.
But, high fidelity explorations do not have to be completely realistic to the current state of technology to accomplish your design or communication goal. Combining elements that are not yet possible but likely to emerge over time is a great way to help the audience suspend their disbelief just long enough to plant the seed of whatever idea you are trying to convey.
For instance, I was reading an eBook out on the deck last summer when I imagined this happened:
Crazy, right? Couldn’t believe it myself, but there it was. The characters of the story appeared right on top of my fire pit when I air tapped that hotspot thingy on the page. Didn’t really appreciate him blasting me, but I’ll take it.
It’s that easy. Just think about a narrative and how you’d bring it to life. Next thing you know, your imagination will have it out there for you to enjoy.
That scene I envisioned in my backyard turned into a high fidelity, photo-realistic rendering of what could be the future of holographically augmented reading. It’s based in reality, but clearly is a fanciful depiction.
Methods
There are many ways to envision, of course. I’ve listed some of the common ways used in holographic below. Can’t say there’s really a “best” way — these exercises are always highly contextual to how much time you have and who the audience is for the final result. Try ’em all over time. You’ll learn something new every time, guaranteed.
For our purposes, these are the most useful ways to envision (listed easiest to hardest, quickest to most time consuming):
- Use your imagination — no work required. Just think and it appears.
- Tell a story —narratives are a great way to make a real connection.
- Act it out — channel your best TV sitcom acting skills.
- Sketching — fast, but not as impactful as other methods.
- Use photos* — my method of choice, and the one we’ll dig into below.
- PowerPoint prototyping- a Designer’s secret weapon; highly underrated
- In-device prototyping — super advanced envisioning technique. Rad.
- Shoot video — super bonus points for doing this one; huge payoff.
- Code prototyping — the best way to bring ideas to life if you can do it
Knowing which envisioning method to choose is one of the skills you’ll pickup by practicing regularly. It becomes pretty obvious after awhile, but that doesn’t mean it’s any easier to execute well.
Regardless, any of the above will help you communicate your ideas to your intended audience.
Rather than try to cover all the previous ways you could share your awesome holographic vison with people, let’s walk through a very specific method that works really well for me. It’s highly visual and quick to execute once you get the hang of it. Just as easy to use this for realistic as it is for fanciful explorations. Welcome to #5 — Use photos.
The “HoloScenes” technique
Based on a series of holographic experience design explorations I did last year called HoloScenes, this rapid envisioning technique uses photos of the real world as its foundational element. They serve as a backdrop for fully integrated people and 3D holograms to get the desired effect for your inspirational experience.
Step 1. Start collecting photos of locations
Taking photos of interesting real-world scenes you come across is the best way I’ve found to jumpstart the envisioning process. Think of it as scouting film locations for your holographic experience (even thought they don’t exist there). Without ever building a prototype, you can use a photo as the basis of a visual sketch of the overall experience you’re after. It’s a setting for your ideas to be showcased.
Be on the lookout for physical places that are calling out for something to be added holographically. Whenever you get that feeling, immediately stop what you’re doing, whip out your phone and snap a picture. Move along.
Sometimes places will strike you as “empty but cool” for some reason. Remember to go back through your pictures when you have time to harvest any that move you. Quickly consider the possibilities. Then share them with yourself or set them aside in a collection you’ll want to grow over time.
I can’t tell you how many photos I have of spaces with no people in them. But therein lies the most interesting paradox in envisioning holographic experiences — you cannot have a successful design without people being the design center. Yet, many of the most interesting ideas come to us when looking at empty space. Go figure.
TIP — using your own photos rather than pulling random images off the Web keeps you out of copyright infringement issues.
Step 2. Pick your scene
Go through your HoloScenes photo collection with an open mind. See what jumps out at you. Try to come up with scenarios that feel like a natural fit for what the scene is telling you could be possible. I wouldn’t try to force a photo into doing a job it’s not meant for— it’ll look off. You’ll see how this works when you try it. The photos actually speak to you if you let them. Go with the flow, and write down some notes on what could work. Set a copy of the photos aside.
TIP- use your strongest photos as the basis for an envisioned scene. No point in distracting the audience with low quality, grainy, or blurry photos.
Step 3. Identify the scenario
What was that first thing that flashed into your mind when you looked at this photo in your collection? That’s the scenario to evaluate first. Or at least pick something similar enough that it feels unforced.
Start asking yourself a few key questions about the scene:
Q1. Where are the people and what are they doing?
Q2. Why would they be doing that with holograms?
Q3. Does this feel like something you’d want to do?
Among my interesting photos, the fancy wooden display alcoves screamed out to me as needing cool content within them. Wasn’t sure exactly what would be best to showcase, but I knew there was a ton of things that could be done with these beautiful indentions.
Artwork seemed like a natural fit.
It also occurred to me that the primary interaction with the content would be through direct manipulation — something you can’t do in the real world but would be safe and fun for the audience. I imagined they would want to touch, poke, move, or speak with the artist to explore its story and details.
TIP — There’s nothing bad about using multiple photos for the same scenario exploration. I would suggest you focus on just one at first to see if you can get your idea across effectively.
Step 4. Sketch the hero
Every epic scene has a hero. Find yours. Heroes are most likely going to be wickedly cool holograms, but they don’t have to be. It could be that your modest holograms enhance a real world hero that’s already there.
Next, we need to do a quick sketch of how the hero will be shown within the scene. Take the photo into an app that will let you scribble on top of it (figuratively or literally). PowerPoint works great for this. Alias Sketchbook is another favorite of mine for this purpose. You could even print the photo out and just draw right on top of it with a fat Sharpie. Whatever works best for you.
Since you have an idea of how the hero will behave, use markup lines to illustrate movement or interaction even at this early phase of sketching.
TIP — Try a few different approaches to depicting your hero, but keep it fast and fluid. It’s nothing to obsess over. Just enough to get the general feel across.
Step 5. Add people to the scene
As we talked about at the beginning of this post, people are the key to any holographic scene. You need to make that human connection with the person you’ll be showing this to. It should be completely obvious why someone would be doing whatever it is within this scenario. Showing people that are present and engaged within the scene makes that easier.
You can use any representation of a person that fits with your scenario and artistic style for this part. I often use the silhouette technique favored by architects and conceptual artists. I find that easiest from the production aspect and it also complements the visual treatment I going for.
You don’t have to be an artist to get that great people effect in your project. Try searching for the type of people you need to add, whether as full color photography or silhouettes (my preference). You’d be amazed how many different body types, sizes, poses, and actions are available. Think about what you’ll need for the current piece — don’t get slowed down by finding the exact perfect set of these. Keep moving!
Once you have the people picked out with the right poses or at least attitude, drop them into the scene in a realistic size and perspective (so we don’t distract the audience with our gaffs). Remember, people are the anchors that make or break the believability of the envisioned experience.
Get in there and distort or twist the people into believable perspectives.
Generate a few large size comps of the scene with your sketchy hero and the people who should be in the scenario. You may have to futz with the angle and perspective of the people you add if you’re interested in a realistic feel. That’s easily done in apps that allow for distorting shapes or adding perspective.
TIP — You’ll be using alot of different people in your envisioning work, so best to start another collection of these, too. We need all kinds of poses — sitting, standing, action shots, doing specific activities, etc.
Step 6 — Swap in the holograms
Now that the scene has people and a stand-in hero, time to add the other holograms that support the activities and scenario. My suggestion here is to use the highest fidelity representations you can find (either as prebuilt 3D models or dimensional photographs), or create your own custom 3D models of holograms to make a high impact first impression with your audience.
The integration and compositing step of this technique is where you can spend a ton of time if you’re not careful. The goal is to rapidly depict what you’re after with fully lit and integrated content — but not at the expense of being done and moving onto other important explorations.
There’s nothing wrong with low fidelity sketches or rudimentary stand-ins. They just lack the magic of a photorealistic hologram (which you are going to presumably build if this exercise is successful).
It does take longer to carefully integrate holograms that look realistic into the scene, but the payoff is worth in — both in your envisioned pieces and also in the final form.
Step back for a moment to see if your scene has already come to life by just adding the hero hologram and people.
If not, you’ve missed something important. Look critically at each part. Is the photo not the right angle? Are the people not the right size or looking disinterested? Does the hero not look like a real hologram? Whatever it is, isolate and fix it so we can move on.
Next, we need to add our supporting cast to the scene. Create or find all of the other holograms that fill out the overall experience. Same rules apply here — use the highest fidelity stand-ins you can find within the shortest amount of time. Take care to adjust their size, orientation, and visual appearance to be just convincing enough to get the job done.
TIP — It’s tempting to go overboard when adding all the detail of the supporting holograms. Go ahead :-) It’s a blast if you’re not super pressed for time.
Step 7. Refine the scene
Once your scene and all the actors are in place, you probably will want to make a final pass to get all objects sized properly, adjust overall lighting, get rid of extraneous elements that don’t help tell your story, and even add some sound (which is an incredibly important part of designing for Mixed Reality). Sound is your secret weapon in bringing ideas to life.
TIP — Don’t overlook adding key sounds to a static image. You wouldn’t want to go see an IMAX 3D movie with no surround sound, right? Same thing here. And easy enough to add if you’re using PowerPoint or Unity for your project.
Step 8. Test it out
The intent here is to literally just show your comps to a few people to get a reaction. With no prompting by you whatsoever, did they say out loud what your intent was when you started the envisioning process? Be honest here. It’s ok if they didn’t get it. Just means you have more work to do.
Even if they did get it right away, I bet they made some suggestion or asked why something was a certain way. Remember those. Go back and fix the scene so it reflects that feedback if it makes sense. Buy that person a coffee and call it good.
TIP — Even in this super early phase of ideation and experiments, you have to be willing to listen carefully to people’s feedback. It’s human nature to not want to hear criticism after working so hard. But, maybe it is true that your holographic unicorn looks like a horse with a fake horn. That’s fine. We’ll get it next time around.
Part 4 — Advanced Envisioning
Sometimes a picture or a story really isn’t enough to achieve the envisioning outcome you are after.
If that’s the case for you, it’s time to break out a tool that can help add some interactivity, motion, and audio to the mix. There’s a ton of those tools available, and their sophistication ranges anywhere from PowerPoint all the way up to a full 3D modeling and animation monster.
Remember that envisioning needs to convey the most critical parts of your experience in a persuasive and convincing manner. Rapid prototyping or coding might be your best bet to achieve that.
Moving to a prototyping mode does not mean that the nature of your ideas has to be completely realistic. To the contrary. You can use prototyping to depict things that cannot exist in this way yet. The key is that your toolset dictates how convincing that can be.
A few things to consider…
Choosing a prototyping tool
In my case, I’ll always lean on a great tool for Mixed Reality development and prototyping to help me figure out what’s what — Unity 3D.
It’s definitely the power tool of choice for people experimenting with VR and AR. You can do just about anything you need to for rapid envisioning from the GUI, but it’s so much more when you write a bit of code for special functionality.
Create a simple representation
You don’t need to exhaustively model and light every detail of your scene to test out an idea. The reason you’re looking into a tool like this is probably to get very specific features working in order to try out the impact on a real person. Build just enough to accomplish that and no more (at this early phase).
TIP — interaction prototypes should be considered disposable. Don’t put too much effort into them so that you don’t want to discard them and move on.
Focus on key issues
Bringing your work into a prototyping environment gives you the opportunity to zoom way in on potential problems and work on realistic solutions early on to help gauge feasibility (something not often discussed during envisioning phases, but helpful to the team to understand early).
In this example, there was a basic question of how close you could get to the painting itself before rendering artifacts would surface to spoil the moment. Using a static image gets the intent across, but doesn’t help answer the question for the team early on.
You can do some of the same type of evaluation in the emulator, but it’s not anything like being there with a real holographic device. Building just enough of a working prototype helped quickly figure out where the problem spots were in the interaction design and asset creation.
Real device, real world
As much as I encourage people to try going the static image route for quick turnaround, there’s nothing quite like seeing your holographic experience come to life with a real device like Microsoft HoloLens. It’s breathtaking. Super fun. And shows exactly what you got wrong.
TIP — learn how to use Unity 3D to compose simple scenes. Then learn to code just enough to get basic functionality like triggering actions, playing sounds, and loading new scenes working for your holographic experiences. It’s makes all the difference when you need a knockout punch.
Final thoughts on Envisioning
There’s so much more to talk about, but for now here’s a few things to end with...
Focus on the people, not the tech
It’s so tempting to center your experience around delivering breathtaking holographic effects or technical marvels that can’t be done on other type of device. Don’t do it. Think about how that signature moment in your design feels to the person experiencing it. Build on that emotional connection. The interaction mechanics will follow and fall into place.
It’s a learning process
Like anything worth doing, practicing envisioning is the key to getting comfortable with and proficient at designing for this medium. You’ll stumble while learning what works and what doesn’t, but that’s the whole fun of it. Try things, make mistakes, learn, do it again. Don’t give up. Your efforts will pay off.
The ability to think and act in 3D will be the new must have skill for high tech Designers over the next few years.
Can’t wait to see what you come up with.
About
M. Pell is a Mixed Reality Envisioneer who leads Design for The Microsoft Garage, the company’s outlet for experimental projects worldwide. Connect with him on Twitter, LinkedIn, Medium, or his website.
Related posts by M. Pell:
“HoloScenes” — envisioning holographic experiences
“Datascrapers” — a data art series
“Fast Design” — the art of moving your ideas forward
“Designers: Adapt or Die” — learn to code or be irrelevant
You can also watch the video of this talk from @WinHUGR on Dec 14, 2016
