Storyboarding for Mixed Reality Enterprise Solutions

Harness the power of visual storytelling & help stakeholders better envision mixed reality experiences

Published in

Experience Matters

5 min readSep 26, 2023

There is a lot happening in the high-paced world of mixed reality technology lately. Between the mixed reality glasses like Microsoft Hololens 2 or Meta Quest Pro and ambitious upcoming projects such as Apple Vision Pro or Meta Quest 3, it is an exciting time to be designing and developing virtual and augmented reality solutions.

As part of SAP’s Mixed Reality Solutions team, I have the pleasure of working and experimenting with the latest mixed reality technology. Our team develops mixed reality applications for enterprise contexts in close involvement with our customers. Because mixed reality is still a relatively new technology for many of our customers, visualizing the concept through storyboarding is an essential part of our process.

Storyboards should be part of your mixed reality toolkit

Storyboards are visual representations that depict how a user will interact with the mixed reality application within their physical environment. They are a crucial component of designing and developing mixed reality solutions because they:

Help to better visualize and plan mixed reality experiences
Support iterative development and testing
Enable effective communication among teams
Identify and solve design issues early
Allow for user testing and feedback
Gain client and stakeholder approval

Below, I’d like to take you through one of the methods we use to visualize immersive experiences. But before I go into details, let me start with the end result: The storyboard!

An immersive storyboard for SAP Intelligent Agriculture. In the storyboard, a farmer named John is seen standing in a corn field accessing a hologram of the app’s UI. In the story, he identifies a pest infestation and sends a drone to the fields to manage the pest. A week later, John again pulls up the hologram dashboard and sees the control of the infestation was a success.

For the above example, I pulled a use case from SAP Intelligent Agriculture. In the area of agricultural intelligence, many new technologies are and will be introduced, such as:

· Autonomous GPS-controlled tractors, harvesters, and robots
· Intensified use of drones for pest control and field scouting
· Networking of data from farm managers with their manufacturers, customers and partners

At the center of all of these technological advances is the imperative to ensure that agricultural workers can dedicate their full attention to the agricultural processes and products. And this is where mixed reality can play a crucial role. I would like to point out that the chosen business case is only an example and does not imply a product realization.

Breaking down the storyboard components

The storyboard should be created at the beginning of the development process, which means you can expect that there will be a lot of changes as the design and development progresses. Whether you choose a comic-like or realistic visual style is up to you, but most storyboards will share the following characteristics:

The storyline

The storyline is the starting point for the storyboard. There will be changes just as you go along, but it’s still a good idea to take your time with the story to minimize rounds of changes. This can be barebones to start. All that is needed is a sequence of images with the story text and a brief description of the content of the image. Also, an arrangement of the images on the storyboard is useful to get the aspect ratio of the images.

A storyboard box skeleteon featuring the caption “John opns his dashboard to get an overview of his tasks at hand. All seems to be fine, except..”. In the middle of the box, the placeholder reads “John stands in front of his dashboard in his cornfield” and no image is yet shown. — *When creating the storyline, it’s easier to work with placeholders and worry about the images later.*

Visualizing the physical environment

To visualize the physical environment and give the mixed reality solution real-world context, I used the new capabilities of generative AI, in my case, the generative fill of Adobe Photoshop. This tool allows you to quickly generate environments. You select an area, describe the content prompts and watch how generative AI generates multiple image suggestions.

In order not to push the environment so much into the foreground, I put a grayscale filter on top of it.

A series of 3 images showing how the author goes from AI text prompt “Sunny day with some white clouds/Corn field with broad green field path in the middle” to an AI-generated image of the prompt, to the final grayscale version inserted into the storyboard format. — *Using an AI-generated image as the physical backdrop for the storyboard*

Arranging UI objects

Once the physical environment is defined, it’s time to bring in the UI elements and user into the story.

To construct the UIs, I used SAP’s Design Stencils for Figma, available in the SAP Fiori Design Guidelines. My own experience with Figma was very limited, but I was easily able to create a farming cockpit dashboard with the content input from SAP’s Intelligent Agriculture team.

I then imported the Figma UI as a vector graphic into Adobe Illustrator, made it slightly translucent, and built it in with correct perspective.

Since 3D models are an integral part of mixed reality applications, I also integrated a 3D drone into the storyboard. The drone was created and colorized in Blender and placed in the storyboard (positioned, scaled, rotated).

The last step was to insert the user. Here I took a photo of myself, masked it, and positioned it accordingly, as shown below.

A composition of various images showing how the different elements — the UI, the 3D drone model, and the user — are placed together against the physical environment backdrop (the farm) within a storyboard box. — *A spatial composition of the user interface, user, and a 3D model in a physical environment*

Mixed reality storyboards are just the tip of the iceberg

With the introduction of mixed reality, it’s time to rethink how we design storyboards and communicate our product ideas to stakeholders. If you compare it with the current approach to 2D applications for devices like laptops, tablets or smartphones, a sequence of 2D screens is no longer enough. Current developments in mixed reality technology and generative AI will change our understanding of storyboard creation even further.

What does the future look like? I can only guess, but maybe I’ll be able to…

describe the necessary components and generate a storyboard.
automatically create an AI-generated video of the application.
even go as far as design and develop a mixed reality solution from my prompts.

We are only witnessing the beginning of these technologies and the results are already very impressive.
In any case, it remains very exciting to think about how to visualize future concepts.

To learn more about Mixed Reality Solutions at SAP, read Mixed Reality Solutions at SAP: What’s It All About?

Special thanks to Oliver Rückert, Solution Expert for SAP Intelligent Agriculture.

Experience matters. Follow our journey as we transform the way we build products for enterprise on www.sap.com/design.