A spatial operating system?

Thoughts about the 3D interface design in VR/AR

Cerebro, Source

When we talk about VR in the sense of that it’s more than just an immersive gameplay platform, we mean it to deploy as many scenarios of the general information service as what is there in the information industry. Be it a sensory revolution or not, VR/AR will definitely change the way we use Internet services and perform tasks. Say, an operating system that is distributed in 3D and manages programs spatially.

With that being said, Microsoft is plugging away at its Windows Holographic operating system to enable mixed reality on VR devices. The holographic platform takes advantage of the virtual world by mixing real people and real environments into it, and scattering 3D web contents all around them. For example, you can grab any digital asset from a distance to you, when you are done just return it back to its original place.

Spatial Cognition

Imagine going inside your room, you know by instinct to sit by the desk to read and write, or take some clothes from the wardrobe. If you are on the street, there is the transportation space of vehicles, the public space for pedestrians, the green space of vegetation, etc., the maps and signs to tell the direction. As a human we already have abundent experiences of interacting with 3D space and a solid correlation with its affordance.

The VR/AR is essentially a digital 3D space. The spatial semantics enable users to do things with direct manipulation and distributed cognition. When a user transitions into the virtual world, he should be able to gain an overview of the space structure and content referring to the metaphor of a real space. Since people have also formed the empirical cognition of 2D computing during last decades’ practice, it is important to note that the we are neither using VR/AR to build a virtual clone of physical environment, nor simply immerse users by packing a 3D background into the 2D interface. Hereby the challenge is that, to Reinvent the rules as Vincent Munoz puts it.

Apple OS X Launchpad and Dock (up, Source) and BumpTop 3D Desktop App (down, Source)
“A user interface where you can navigate in 3D can enrich the communication between humans, products, companies and information. A new world of opportunities is open: Interactive menus with spatial configurations, 3D webs with unexplored topological organizations adapted to the full scale and motion of our body and to the cognitive potential of our brain, and above all, a more seductive, sensorial and emotional user experiences.” (Nacho Martín)

Natural User Interface

Pinch VR UI (up) and Multiple Monitor Display (down), Source

This is probably the most common VR UI layout that is applied in multiple devices. The advantage of this approach is that it curves the GUI around the users and make it easy for them to see and operate. It sort of reconfigures the idea of multiple monitor display and qualifies it for an immersive experience.

Though I’m not sure GUI is the perfect user interface for VR just because people have ramped it up in PCs and mobiles. A natural user interface is by all means integral to the its environment and become seamless and invisible. In VR/AR space, people don’t have a perfect sense of coordination system given too much degree of freedom, losing spatial mapping hence takes them out of immersion and causes frustration. Depth cues of GUI in VR environment is yet difficult to represent. There is no stereoscopic 3D effect on flat screen-based GUI, and we are unable to apply light, texture, parallax, or other visual features on it to convey the spatial attributes.

Some tools to implement for making the 3D interface more natural:

  • Greyboxing
Alex Chu — Transition from a 2D to 3D design paradigm, Source

Define a space field as a basement for 3D user interface, give user a perspective on the overall structure and limit the degree of freedom. It will also be possible to customize the space settings to guide users through the experience.

  • Diegetic interface
Virtual Desktop, Source
Microsoft’s 3D Desktop Prototype, Source

Embed the interface seamlessly in the virtual space to help users maintain a sense of space and scale. A lot of potentials here would be empowering interactions with geometries instead of polygons. For example, a cubical bundle of interface like this:

KDE 3D Workspace, Source

The gaming-derived interface has the great benefits of creating consistent and realistic experiences. By virtue of engaging with the Hovercast VR menu (triggered by gestures) and other forms of hologram, diegetic interface allows for providing diverse interaction modes within different scenarios.

The CAVE (left) and the CYBERSPHERE (right), Source

Similar to the CAVE and CYBERSPHERE environment, the 3D interface might even construct a container-like volume surrounding the user with interactive contents, in other words a sub-space inside main space.

  • Dynamic 3D visualization

Motions and animations to explain the interaction have been extensively implemented in 2D computing. VR/AR as a spatial medium can enlarge their effects to a great extent. In Jody Medich’s article she demonstrates this idea of leveraging dynamic movement between data visualization to reveal data patterns. Therefore, users are better informed of the context when transition happens.

“Data is given a dynamic volume as defined by the values of the intersecting graphs. By quickly looking at the dimensions involved, the user can instantly understand the difference between various items.”
Dynamic Dimension: ABC x Time (Jody Medich, Source)
Dynamic Dimension: Recency x Frequency (Jody Medich, Source)


Here shows how multitasking works in 2D computing:

Apple OS X EI Capitan Mission Control, Source
Android (left) and iOS 9 (right) App Switching, Source

The painpoint of switching between tasks lies in that it breaks the workflow, causes distraction and memory loads, which leads to low productivity and efficiency at last.

A 3D immersive system, however, displays an unlimited workspace in front of the user. The virtual space keeps reshaping and sprawling as the user navigates around, making it vastly customized. Note that there’s a massive productivity gain when people have the possibilities of spatially organizing their tasks and files.

When it comes to the presentation of an app, the system serves users with a highly intuitive information structure and navigation path (visualized in a style of 3D sitemap), as well as facilitating work-in-parallel and alternative views.

The spatial distribution of user interface also contributes to avoid further objects being occluded by closer objects.

Prezi path, Source

Prezi is an excellent example to help us get a glimpse of spatial arrangement in VR environment. By panning, zooming, rotating, and orbiting (not in Prezi), users are allowed to display and navigate through information assets. Paths are navigational sequences that logically thread all the objects. While the presentation trail in Prezi is linear, 3D interface will notably constitute more reticular connections.


VR/AR brings the potential of spatial working environment, which shifts the user interface design from 2D flat to 3D spatial. A 3D interface employs users’ spatial cognition, strengthens natural experience, and improves multitasking.

* References:

  1. VR Design: Transitioning from a 2D to 3D Design Paradigm, by Alex Chu
  2. What Would a Truly 3D Operating System Look Like?, by Jody Medich
  3. 6 Principles of Leap Motion Interaction Design, by Alex Colgan
  4. Introducing Widgets: Fundamental UI Elements for Unity, by Andrew Littlefield
  5. Designing VR Tools: The Good. the Bad, and the Ugly, by Jody Medich and Plemmons
  6. Game UI Discoveries: What Players Want, by Marcus Andrews
  7. The VR Book: Human-Centered Design for Virtual Reality, by Jason Jerald
  8. 3D User Interfaces: Theory and Practice, by Doug Bowman. Ernst Kruijff, Joseph J. LaViola, Jr., Ivan Poupyrev

Thank you!

Lang Tian