This post is an adaptation of a presentation I had the immense honor of sharing as part of the Mixed Reality track at Interaction 20 in Milan on 05 February.

Before I dive headlong into my creative process for creating spatial apps — I would love for you all to indulge me in a small exercise. In your mind — describe the size of the group you are sharing your current space with — don’t spend a long time — what is the first thing that pops in your head.

If you are in a conference with a gaggle of folks or even working in a local coffee shop, you’re probably hearing very factual statements — 5 people, 10 people, less than a 100. These are very straight forward numerical statements, now what happens if we add only a handful of multi-dimensional relationships? How many are wearing glasses? What is the dominant color scheme of their clothes? What about hair length?

We can quickly see how hard it is to communicate multi-dimensional relationships. We use things like similes, metaphors, and abstract statements to try to summarize the audience’s size now.

This was just a quick demonstration of when things are more complicated; they take longer to expound upon. It also demonstrates human’s unique abilities to quickly grasp and rely upon abstract concepts to communicate complex ideas.

My team at IBM’s client experience centers communicate and explain the highly technical and complex ideas behind AI, quantum computing, and IBM’s innovations to clients, researchers, and students.

The IBM client centers use a 290° immersive room to place visitors in a communal XR experience. Our expert-led experiences leverage 93.3 million pixels to demonstrate AI interacting with data at a scale that allows humans to see how machine learning makes decisions, how it interacts with data, and the value it holds for their organizations.

You’re probably asking yourself — when is this fella getting to the creativity stuff? Let’s refer back to those examples of abstract and concrete statements we demonstrated earlier. I promise — I keep going back to these with a reason.

The creative process is your unique ability to think abstractly. Creativity is not high science, there isn’t a magic incantation, there isn’t an app you can buy — and despite every Medium article saying otherwise — there is no hack or shortcut. Every design shop, creative agency, or innovation consultancy follows the same creative process — just in different hues and colorways.

IDEO calls this with Divergent (abstract) and Convergent (concrete) thinking. IBM calls these steps in the Loop Make (abstract) and Observe (concrete). Our group creates explicitly experience about our products and tech; we take the same approach but balance it with a story (abstract) and facts (concrete).

Progressively shifting your mental models from abstraction to concrete thinking and moving forward linearly is the creative process. Creativity is a conversation — expansion and contractions, story & facts, abstract & concrete. As you continually progress from abstract to concrete, this process can run to a finite point or continue indefinitely. The process is not prescriptive — it is merely a sandbox to focus your creativity.

Assimilate, create, iterate, repeat ∞

This is the exact same process we use to guide our experiences, but we constrict the time frame greatly. We smash in a week of intensive and utterly exhaustive workshops and co-creation sessions. That same formula — shifting from abstract to concrete thinking is the exact methodology of a design workshop — that’s why they are utterly exhausting. You condense the process and activities that recreate that conversation between abstraction and concrete thinking into formulaic steps.

We partner with internal technical experts who co-create with our team for a week — we deep dive into how products work, the technical underpinnings, and unique differentiators. By the end of the week — bleary-eyed, exhausted — we are finally iterating, prototyping, and experimenting at scale in our lab in Austin.

We use tools that we can quickly spin up visuals withs — things like Processing/P3, TouchDesigner, some C++ explorations in space — even our sketches in a pinch. We’ll throw them on our 290° wall to get a sense of what it will be like at scale — low tech and easy to do — the quicker, the better. The important thing is not letting the technology dictate the craft, much the opposite, the tech is just a tool. We are focused on story, flow, clarity, and interaction.

Create tension

Our group develops explanations of IBM innovations and technology. That creative process we detailed before comes back even within our approach to storytelling — creating compelling stories that are a conversation between humanity and technology. We don’t tackle these tasks independently, but for the sense of the presentation, I’ve broken it out to demonstrate how we create product stories.

Product stories are primarily driven with obviously the product and outcomes. This is our bread and butter — but what is that old adage…

The only thing worse than a bad story is a boring one.

But between products and outcomes lies the crux of all great stories — challenge. Good vs. evil, friend vs. for, user need vs. product. Not a diametric fight but a give and take — every good story or product needs tension — or balance. Challenges are universal if you communicate them elegantly. Our experiences are viewed by everyone from an expert to a Luddite. By expressing problems and humanizing them — we put every audience member on a level playing field.

We lead all of our experience with introducing the challenge that the team or product was forced to overcome and with our introduction — be it a product or technology and our outcome — now we can create story beats or actions that string that story arch together. In our work, we like to have a sense of evergreen in it — if a visitor returns to the centers, we don’t want them to see the same thing. We use that story outline to create a decision tree to create a stateful app that tells a multitude of linear stories — much like choose your adventure games. This allows us to develop heightened tension and make the stories approachable to all our visitors.

Design your verbs

Using that flow as an outline, we can now start to align either product actions, user inputs, or data insights as story beats that help to progress the experience. We can’t just put up screenshots of our products to tell the story of our tech — we are trying to give visitors a look inside the black box of AI, so we need to break down the product and present it as a story that demonstrates it’s value. We offer the true nature of the product and use real-world data to prove its tech.

So we’ll take something like prioritizing alerts with AI and present that in context using real data in the immersion room. We’re demoing distinct features of the product demonstrated with 1000s of anonymized financial transactions in real-time. With a gesture, we can identify the cases with the highest priority in real-time. We take the specific aspects of the product and using the technology to help clarify what it is doing and how it works.

If we take a look at another step in this product flow: AI-assisted report creation. We can take something germane and make it more compelling and a strong example of AI. Following that same creative process, we iterate on a single idea — from workshop sketches to testing interaction wireframes, to exploring inspiration concepts, to refining a visual design, until our final design artifact represents AI report creation.

AI as a
design input

We even use AI as a partner in the creative process when we create bespoke visualizations for clients using a combination of publicly available data, AI, and generative art. These pieces act as a simple example to clients how AI can transform data, and for the creative team aids in the creative process as a form of visual abstraction. AI will help you to iterate quickly to produce visuals that are driven from data — where every color, scale, and rotation has hidden meaning that serves as a customized art piece for clients.

Art resides in the quality of doing — process is not magic.
Charles Eames

This quote serves as a constant inspiration to me. The unique thing about humans is we are all born inherently creative, but staying original is hard because we are pre-wired to find shortcuts. The creative process is so hard cause it forces you to push and iterate until you are satisfied with the results — regardless of time and regardless of burnout. You have to be consistent, and you have to be steadfast — as I said at the beginning — there is no magic bullet.

But that quote inspires me in another way as well. I’ve found when I’m struggling the most with a project or overwhelmed by tasks or projects are not progressing as planned — the best thing I can do is make anything. The act of doing — acting with purpose and free of constraints allows me to overcome those challenges directly.

I guess if there is one more thing I would love to impart to everyone, is that — make — do — create. All of us — my self included need to not be as precious about our craft and creativity. We shouldn’t be concerned with what prototyping tools people are using. What programs are best — I’ll be honest — the project that mattered most to me at the end of the day — my design babies, all started with my sketchbook. I have miserable handwriting, and my drawings are scribbling that only make sense to me — but this is where I can be expressive free of constraints.

I don’t need to know C++, I don’t need a +$7000 cheese-grater pro computer for this. All I need are me and my creativity — the human brain is the best GPU you can buy — use what you have to create.

Thanks so much for IXDA for letting me chat about this and thanks so much to all the amazingly talented creators in the mixed reality track. Everyone shared fantastic work, and I would urge you all to check it out.

Courtney Sheehan spoke about the emerging tech in dream research, lucid dreaming, and shared some insights from their upcoming documentary. Karen Stolzenberg from Magic Leap shared some amazing work from the Magic Leap Social team. The most intriguing part I found to be their work focused on creating shared but individually contextualized spaces for VR interaction. Ana Sofia Gonzalez shared insights around creative transition to working in 3D and how to translate content to mixed reality devices like the HoloLens. Marie Jasmin spoke about defining “Time well spent” in gaming and how pushing users to be slightly uncomfortable can create memorable experiences. They also worked on Assassins Creed 2 and designed the DNA UI, which I spent WAYYYYYY to much time using back in the day.

Grazie mille. Cin cin.

Rob Harrigan is the Design Principal of Immersive Design & AI Services at IBM based in Astor Place. The above article is personal and does not necessarily represent IBM’s positions, strategies or opinions.

This article was originally published on my personal site and is available without paywalls.