The definitive guide to crafting engaging multisensory experiences in a physical space with the use of emerging technologies

Marina Razakhatskaya

Follow

Apr 29, 2019 · 8 min read

As Creative Technologists, we spend a lot of time working out how best to communicate complex ideas in an engaging way. While tools might range from web, mobile development and experience design to hardware prototyping, wearables, creative coding and art installations, the brief for this type of “storytelling challenges” is often the same:

“How might we communicate [ __________ ] without slides?”

While storytelling has always been immersive (think theatre, for example), technology gives us tools to unlock additional senses, boost users’ imagination, spark dialogue, and fuel self-expression.

The evolution of technology has shaped the way we design, build, and consume products, services, and experiences. Let’s look at the gaming industry, for instance.

We started with sports and board games before we were introduced to video games. We moved from TV screens to online gaming platforms on our smartphones. And eventually we found ourselves immersed in 3D environments via VR headsets or outdoors chasing Pokémon in AR.

For the past decade, the design industry has focused primarily on screens. We mastered our craft building elegant digital products that aim to make our lives easier, more connected and more productive. Recently, however, we have started to see the unintended consequences that this digital transformation has had on how we live. It didn’t take long to see design respond, and we soon moved from the “You’re All Caught Up” feature on Instagram to the “We’re humans, we’re taking our lives back” vision behind the Light Phone product.

Physical space becomes the new canvas as we seek more engaging tangible experiences packed with magic.

The craving for multi-sensory experiences with physical objects and spaces has led to the boom of ‘Instagrammable museums’ — Museum of Ice Cream, Color Factory, The Dream Machine, 29Rooms, etc. — that have used the journey of discovery and fun as a new business model.

Our expectations of space have also evolved as a result of digital transformation. Mobility, transient living, and the gig economy — all enabled by digital — set the demand for multi-purpose spaces, plus space-as-a-service and space-as-an-experience offerings. We see retailers, workplaces, and the hospitality industry being the early adopters of what Fjord’s 2019 Trends report dubbed ‘Space Odyssey’. New York became the home for Nike’s House of Innovation, where online and offline shopping experiences blended together and the space itself became an experience, filled with craft, activities and objects of delight.

As a creative technologist, I look at new tech as an enabler that makes the physical world interactive. We often complain about screen fatigue, but rarely about click fatigue. This is simply because — as much as we want to get away from screens — we don’t want to compromise the convenience of getting instant feedback from our interactions.

And if we look at emerging technologies today (AR/VR, computer vision, IoT and AI algorithms powering those from behind), we’ll see that all these technologies have advanced for the same reason— to connect physical and digital.

While technology enabling interaction in physical space is already there, real-world applications are still in an early phase and are often used as a WOW-factor. The evidence is found in pilot projects like Amazon Go, in conference demos at SXSW and in pop-up experiences and brand activations.

One of the factors slowing down adoption is the lack of tools that would enable designers to design for blended physical-to-digital reality with emerging technologies. This is why the role of creative technologists — designers who are hands-on with technology — becomes critical and in high demand within innovation hubs and experiential agencies. Apart from tools, we also need to develop methods and frameworks for designing interactive experiences in physical space.

While designing technology-inspired and technology-enabled immersive experiences at The Dock, I’ve identified five key components that are critical for building interactive experiences in a physical space. These components form a framework that facilitates the design process and can be applied to any space, audience, and theme.

1. Installations

Installations have the power to transform a desert into the largest open-air art gallery in the world during Burning Man festival. In physical space, installations are the highlights of your story, the milestones of user journey.

Interactive art installations are powerful tools to communicate a complex concept in a fun and engaging way. They serve as opportunities to demonstrate a new technology or product or service in a metaphorical way and inspire thinking of real-life use cases. This is why installations are now being taken out of art galleries and find their new niche in retail and work spaces, in brand activations, and in centres for corporate innovation.

Installations replace slide decks and use means of indirect metaphorical communication where the message is hidden and needs to be unwrapped as a gift. On a human level, installations trigger the excitement of discovery and make users work/engage more to get the message. And when we put effort into something, we tend to value it a bit more than a given.

To get some inspiration of how interactive installations might look and feel like, check CreativeApplications.net.

2. Intuitive Navigation

The paradigm of design navigation for 2D digital products is well defined. We have UI components, clicks, scrolls, taps and swipes.

When designing for a physical space, we need to consider a custom 3D environment where navigation can barely be standardized. In a physical space, the design of navigation system depends on a large number of variables:

• size of the space
• time of an experience
• whether it’s a guided or self-discovery experience
• whether the journey is linear or can be experienced in random order
• whether it’s an individual or group experience

Navigation systems are crucial in enabling users to experience an interactive physical space the way it was designed to be experienced. It can be designed using one or a combination of the following tools:

• signage system (think road signs, emojis, etc. as an example)
• floor markup (stickers, carpets)
• light systems (interactive lights switching on and off in a dark room guiding users through the journey)
• audio system (navigation guidance enabled by audio recordings or sound manipulations)
• human guide (think museum guide, workshop facilitator, actor)
• gamification (treasure-hunt game principles to move users from point A to point B)
• and focal points (intuitive navigation enabled by scenery design).

Maybe we can even learn from designing for VR and apply these learnings to designing for physical space?

3. Immersion

Immersion is about creating a feeling of being somewhere else, in a certain environment or in a certain context.

The word ‘immersive’ has gotten a strong association with VR and AR these days. But it’s much broader than that.
Immersion can be created using a wide range of technologies — or no technology at all. In Black Box Bellagio Data Casino, for example, immersion is created by casino metaphor in space and experience design only. Another example has been found in our “The Future Home” research, which is focused on a human-first approach in designing a smart home. For some of our respondents, being surrounded by personal items such as blankets and candles evoked a sense of safety more than security cameras.

Immersive design is based on triggering patterns in associative memory that can be achieved through sensorial perception. Here we work with senses: vision, hearing, smell, taste, touch, and even sense of temperature. Accordingly, immersion is usually created with set/visual design, light and darkness, projections and projection mapping. It also involves sound and haptic feedback design elements to give users a reference to a specific environments.

4. Interactions

Apart from touch screens, which can be an equal part of an experience in a physical space, the variety of possible interactions with the space is much broader. Input can be taken from:

• full0body movement
• gesture tracking
• eye tracking
• computer vision
• voice
• touch (see below)
• and any source of data that can be captured by sensors.

Touch gains even more importance, because with Internet of Things and capacitive sensors, we can make pretty much every physical object today responsive to touch. Think banana piano or Wishing tree as simple examples.

Feedback systems can also consist of multiple elements such as 3D projections, interactive data visualization, mechanical and haptic response, sound and even artificial smoke. The potential forcreativity is endless.

5. Souvenir

Of course, if you take someone on a journey with you, you want to make sure they are rewarded and come home with a takeaway.

Immersive experience is limited by time and space, and once your audience leaves the room, they start forgetting. This is why a souvenir is a key component of any experience. In this case, it’s not swag — it’s an object to contain the memory of the experience.

What makes a strong takeaway? Think personalization, customization, and emotional connection.

We normally use 3D printers, labels printers and laser cutters to create something unique — ideally in real time in front of theuser. The way it’s presented can be an emotional moment too. Maybe your guest is approached by a mini robot who opens its heart to them, and there is a little gift? Or maybe you ask your guest to sign a letter of intent, saying they’re going to use their data responsibly?

Whether you use technology or pen and paper, there are numerous ways to make a souvenir special.

These 5 key components [installations, navigation, immersion, interactions, souvenir] form a design system that helps convert any physical space into a tangible, immersive experience that might cause delight.

If you think this might not be applicable to your business, service, or product, think again, because according to Vice Media,

“8 out of 10 young people would rather spend their hard-earned money on an experience rather than a physical purchase.”

Even if space is not part of your business model, this trend still can be embraced in a form of pop-up experiences, brand activations, client pitch sessions, and trade show experiences. And if you don’t know what to start with, feel free to use the above framework.