Inside a Microsoft Design Experiment
Bringing dimensionality to everyday applications across devices
Dimensionality. It’s a design concept that’s near and dear to the Microsoft Design team’s heart, especially as we continue to evolve our Fluent Design System. It used to be that dimensionality — that immersive, interactive feeling — was expected in gaming, end of story. But now we’re asking: how do you embrace that experience in everyday applications, engaging people on any device? A small team of our designers, technical artists, and app developers used the Universal Windows Platform and the humble weather app to answer that question.
Spoiler alert: it was a challenge. It continues to be a living experiment in the realms of scalability and transcendence: we’re learning, tailoring, fine-tuning, failing, triumphing, and learning again. Here’s a glimpse into what it means to build across dimensions.
Building holographic
We chose HoloLens as our starting line, building a 3D experience in mixed reality that would be easy for newcomers to grasp and enjoy. From there we’d figure out how to bring that experience into a 2D application. The whole idea being: let’s invite people in; use the same components across experiences to build familiarity and responsiveness and a sense of place.
Why a weather app? Turns out you can have a lot of fun with it and build something super engaging in a short timeframe. Plus it’s based in reality, knowable to everyone, and gave us the chance to research some cool places. We focused our attention on phone, PC/tablet, and HoloLens, and flew from there, completing work in a few weeks that explores different ways to bring dimensionality and delight into a universal experience.
The project kicked off in May, a damp and chilly time here in the Seattle area (shocking), so we had sunny, coastal locations on the brain. We experimented a bit to find the right scale and level of detail to build four atmospheric dioramas in HoloLens: Tuscany, Mykonos, Cape Cod, and Monument Valley. The experience opens on the edges of each scene: a slice of a miniature world dropped into your everyday environment. From there, you’re invited to interact with the objects — becoming true scenes that adapt as you move closer and walk around them. Our operative theme here is engagement. You start at the level of information architecture: what is the temperature in Tuscany? Then, we invite you in. Come get closer to the terrain. Now you see the light and the atmosphere. Next, the detailed motion and spatial sounds. Now, narrative details. Someone waving at you, trees swaying.
Because you’re in a mixed reality HoloLens, it’s possible someone in your real world will walk through your hologram. And you will absolutely scream no! as it happens because now, you’re living this environment. It happens fast: you’re feeling something, you’re in it, and you’re emotionally connected. It’s sort of like getting lost in a snow globe. A dynamic, mesmerizing place within a place.
To get the right size, scale and placement of objects, we relied on our 3D artist to work closely with the designers in Maya (3D animation software). This is where we did the blocking to determine the right size, scale, and placement of the objects within a larger scene, just like directing a movie. Each element needed to be sized just so to maintain realistic proportions regardless of viewing distance or angle.
We got the best results — and moved quickly — when we baked shadows into the textures. The realism of each scene came from careful attention to detail and an artful eye. For example, the distinct size, direction, and movement of the waves helped differentiate the Atlantic coast from an island in the Aegean Sea (details matter!). The transition between dioramas is relatively fast and simple, keeping your focus on the scene and the feeling it brings, not the motion itself.
What else makes an appearance? Sunbathers. A sailor. Seagulls, cows, a tractor. We had a lot of fun telling the story and inviting people in. We added subtle animations and learned that polygon count, texture size, and the number of controlling joints all impacted the animation’s smoothness. Ambient sounds (crashing waves) and distinct sounds (moo!) were the final touches to bring everything to life.
Scaling it to 2D
So we created a pretty, fun, interactive thing for HoloLens. Now, how to scale that back to phone and tablet, and make 2D and 3D content that co-exists naturally? Could we reuse the hologram within a flat UI and still make it feel engaging? We’re fortunate that the Universal Windows Platform makes this easy — check out this detailed how-to in another sample weather app, Atmosphere.
One-step conversion is still a dream, and we had to do some manual tailoring to make the diorama look great on 2D devices. We adjusted the shaders for the lighthouse beam to retain a realistic look, and turned off the MIP map compression for the diorama textures. We also experimented with colors that matched the diorama before landing on a grayscale background.
Adaptive input
The Fluent Design System is all about translation and cohesion. Our goal in this experiment was to make each element translate in a way that’s appropriate for the type of device and primary input (touch, mouse, gaze) being used. We designed to maintain the continuity of the experience from both visual and interactive perspectives, while making sure things worked naturally and familiarly on each device. We wanted the HoloLens experience to stay true to holographic and 3D form, but still be approachable and easy for new users — making gesture, gaze, and voice as familiar as using a PC. The 2D experience needed to optimize for the available screen real estate and feel natural with touch and mouse interactions, while still feeling as immersive as mixed reality. Designing outside the box, within a box, as it were.
Here’s a glimpse into some of the work! Take note of the button to switch between scenes — a common design element for all three form factors to help with wayfinding.
The mixed reality interaction was full of volume and scale experimentation — like bringing the switch button closer to the viewer. But not everything benefits from volume; we kept the type flat for readability:
In the 2D experience, we used some of the new Fluent Design System ingredients to help bring continuity while optimizing for the experience on that device — things like light, depth, and motion bring things to life even in a flat space.
Here, a physical sheet of acrylic shows the effect that inspired Acrylic:
And here is Acrylic applied in the weather app to display the detailed forecast while keeping the context of the diorama in subtle sight.
In the phone experience, where a smaller screen size means separating the navigation elements and the content, connected animation gives a continuous, cinematic navigation that echoes 3D immersion.
Here’s what we learned from this experiment:
- If, like us, you are new to developing for HoloLens and want to experiment with an app of your own, the mixed reality design guidance and the Mixed Reality Design Labs code samples are great resources to help you get started.
- Crafting four beautiful 3D scenes and implementing them into the HoloLens was the most labor intensive part of the project.
- The 2D portion of the app came together really fast with the Fluent Design System building blocks.
What we’d like to investigate further:
- The button panels we used in this experiment have coherence and familiarity across 2D and 3D, but might not scale to a mixed reality weather app with dozens or hundreds of locations. With a larger location list, we would probably need to move away from the familiarity of the 2D button panels to something more optimized for mixed reality. What would that look like? Are there other navigation patterns in 2D that would make a smooth transition to a mixed reality experience?
- What other ways can we bring the delight of the 3D scenes into the 2D experience? What are the intermediate steps between 2D and mixed reality, both for a user and for a developer?
Shout out to Creative Director Ramiro Torres, Oscar Murillo and the intrepid team for their work on this app. Next time we’ll add that shark.
We love hearing community feedback. What do you think of our little weather experiment? What else would you like to see us explore? What have you built using Fluent Design, and how’d it go? Reach out to us in the comments below, start a conversation in our LinkedIn group, or hit us up on Twitter @MicrosoftDesign. Can’t wait to see what you create.
To stay in-the-know with Microsoft Design, follow us on Dribbble, Twitter and Facebook, or join our Windows Insider program. And if you are interested in joining our team, head over to aka.ms/DesignCareers.
