Interaction design in VR — a paradigm shift
One of the earliest known examples in the VR space is the Nintendo Virtual Boy, a portable video game console that was a big commercial failure. Although research in VR took a backseat later on, it has slowly started creeping back into our lives. Here, we discuss about interaction design in virtual reality, various prototyping methods and how VR by itself functions as an efficient prototyping tool. The context of existing research in virtual reality helps the reader to understand future work and how VR can be used effectively in design processes for various domains.
Often people see a great overlap between virtual and augmented reality. Both augmented and virtual reality seem a lot similar in the goal of immersing the user, though both systems work in different ways. Although technology in virtual reality is just stepping up the plate while augmented reality is already a commercial success, with the rise of Oculus Rift and Google Cardboard it is going to be the future of social encounters in the virtual world.
Virtual reality is about immersing users in a virtual world such that they are unable to distinguish between what is real and what is not. Here, users are isolated from the real world and fully engrossed in the virtual world. VR is meant to enable people interact with a whole new world built from imagination, or the known world from a different perspective. Augmented reality is the blending of virtual reality and real life. Here, users interact with virtual content in the real world and are able to tell the difference between them. AR might have more commercial success as compared to VR because users continue to be connected to the real world while they are interacting with the virtual world, but VR has been picking pace in creating innovative solutions to design problems, especially as a prototyping tool.
Furthermore, VR creates visceral empathy for many problems that are even hard to imagine. It is intense, emotional and makes people really feel the virtual objects in the environment around them. VR is a great solution for simulating an interactive environment through a fast iteration cycle and low investment. It brings digital experiences into the 3D world.
This article discusses the different interactions possible in the VR space, and prototyping approaches to build VR objects and environments. It also focuses on prototyping for VR and prototyping with VR, its advantages and disadvantages.
Although virtual reality is the talk of the town, it has its downside of detaching user’s senses from the real world creating difficulty to comprehend. VR is always experimented in guided and controlled settings with the use of HMDs (Head-Mounted Displays) like Google glasses or Oculus Rift. These scenarios must ensure that people are safe when their senses are split between the real and the virtual worlds. Disorientation and visually induced motion sickness are well-known side effects of VR.
INTERACTION DESIGN IN VIRTUAL REALITY
Virtual reality technology affords limitless possibilities for interacting with the immersive world. Virtual Reality technology marks the rise of a new medium and presents a new set of challenges for UX designers in terms of interactions and experiences. UX experts earlier designed forms, websites and smartphone applications. It is now time to re-learn the trade skills to design interfaces and interactions in virtual reality.
Interaction is key to VR experience. The element of interaction is very significant to make the user feel immersed in the virtual world. Real-time interactivity — the ability to detect user inputs and modify the virtual world instantaneously helps users connect and fully engage with the experience. The VR world offers various interactions — not only 3D images, but also 3D sound, artificial smell generation, force-feedback and action (immersion). Heim defined VR with ‘three I’s’ (7) — immersion, interaction and information intensity. The virtual world sends constantly updated information to users supporting immersion and interactivity. This knowledge transfer is referred as information intensity, which is the third I of VR, after immersion and interaction.
Interaction can be defined as the process that enables the receiver (user) to take the role of the sender and control the medium. In simpler terms, interaction in virtual reality is the ability of the user to move within the virtual world and interact with the object in that world. Here, the user is immersed in a 3D world and hence this interaction is referred to as immersive interaction.
Designing for immersive interaction can be challenging and offers designers opportunities to stretch beyond existing technologies and explore innovative solutions to make VR experiences more real. Designing for VR is remarkably different and knowledge in fundamental aspects of human perception and cognition will help design VR applications on any platform. The following are a wide variety of interaction use cases offered by VR:
- Role of ground: The ground to horizon relationship is very important in VR as in our physical reality. This cancels any kind of motion sickness associated with the experience. The users must be able to relate and have a sense of orientation in the virtual world.
- Atmosphere: Aerial or atmospheric perspective helps users to scale and design VR environments that make the experience more natural. Concepts of depth and distance become more evident with this perspective.
- Terrain features: These form the ground environment that enable locomotion of objects in the VR space. Using these features in a controlled manner will result in VR experiences guided by human intuition.
- Approach to the environment through soundscape: Change of environment in virtual reality causes sickness to users although there is not much technical difficulty in doing it. A gentle introduction or transition to a different environment can be done with the inclusion of an ambient soundscape at first, and then the image. This allows time for the user to build a mental model of the environment before s/he is immersed into it.
- Wayfinding with objects: Although having a number of elements in the virtual environment might break the immersion of the user, it is ideal to guide the user through an environment with the help of conventional UI elements like a stream of flowing water to draw attention of the users to the correct path. However, the use of such elements must be based on context. For example, the above said element cannot be used on a Martian landscape.
- Contextual reticle: In non-tracked VR like Gear VR and Google cardboard, reticles are used to show users specific points that they are gazing at. It helps for better orientation in space by showing the focus, and can also be used for movement for interacting with objects. Different states of the user like idle state, movement state and interaction state demand different interactions from the reticle.
- Interactive objects: When users are in the virtual environment, the objects that they can interact with, should show up with some hint or with the help of the reticle, with a minor shading in the color of the object or even a subtle sound describing its behavior.
A lot of focus has also been laid on the design of buttons, which is especially tricky in the virtual environment when there is no hand tracking with the hardware. A lot of literature revolves around improving this experience by the use of fuse buttons, representing their hover state etc. Researchers from Google’s Daydream project talk about most common interactions that users perform on VR prototypes like actions of throwing, zooming, tapping, grabbing and moving objects. This paper does not delve into details of interactions by with specific elements of the virtual environment as this creative opportunity can be endless. Instead, it focuses on providing an overview of the type of interactions that VR offers and their scope for use. However, the following section talks about an approach to design and prototyping elements of the VR environment and the environment itself.
PROTOTYPING FOR VIRTUAL REALITY
The technology of virtual reality has been gaining traction. However, there is an absence of methods and best practices of designing it. The medium is fairly new and hence there are very few tools that can be used to prototype VR design. Oculus Rift with extensions from LeapMotion, Gear VR and Google Glasses are existing hardware technology that assist VR experience and prototyping in a very limited way.
Unity is one of the best available tools for prototyping VR ideas. One can place a few shapes in the scene, press play, put on the Oculus Rift and immerse into the scene in a matter of minutes. Modifications or tweaks to the design can be made on the PC and one can go back to see these design changes putting on the Oculus Rift again. Unreal Engine is another such option consisting of game development tools. It allows for VR with custom lighting, shading and cinematic systems.
For a long time now, designers have been building 2D elements to represent objects in the virtual world and this nearly ruins the immersion that VR offers. VR products must have smart interactions and must tell a story. Designing for VR currently offers the following challenges.
- The design space is different from the preview space: Designers create elements on a 2D screen (PC), use a HMD to view how it looks like in 3D or VR. If the design requires changes, the designer must go back to the PC to make required modifications. This process involves designers to go back and forth between the 2D and 3D platforms.
- The design mode is different from the preview mode: The designer must switch back and forth between the 2D and 3D screens. This reduces efficiency of work as his senses are constantly split between the two worlds.
- Prototyping using the currently available tools requires prior knowledge in 3D animations, game design etc. This means that the designer cannot fully immerse in prototyping if he is not adept with the tool. He may want to hire a person to prototype and in this process of transition, ideas often get lost. Also, the designer would be using tools that are suitable to create and design games, but are obsolete for VR design. They may be well integrated for VR, but cannot be used to create VR experience.
With this background, there is a need to find a new paradigm approach in designing for VR, because it should just not mean transferring 2D practices to 3D. Designers should expand expertise to various related fields like architecture, sound design, lighting design and physics, psychology etc. to be able to have a holistic approach towards creating useful and controlled VR experiences. The best way to prototype for VR, or for any interaction for that matter, is to use a tool that is designed for VR and works within VR, as it saves time and is easily comprehendible by users.
Rapid prototyping in VR ensures that designers build the right product iteratively before investing time and money. Fully constructing a test environment for VR can be expensive and time-consuming. Hence creating a virtual version of the environment with high fidelity using VR is a potential alternative that saves time and money. VR prototyping is being adopted by several industries including the automobile industry to realize lot of efficiencies and validating designs ahead of time. Virtual prototyping offers capabilities of alteration in any which way before building the actual product. It allows for flexibility and also helps to visualize the big and small details of a product.
There has been limited literature that concentrates on alternative prototyping for VR, for example, low-fidelity or paper-prototyping. Paper origami over just plain paper for rapid prototyping helps in representing 3D models of objects in VR space. Paper models accompanied with tape, color film or translucent plastic can be used to analyze how users interact in the 3D or VR world. These interactions can be translated into actions in the VR space. Google’s Daydream Project encourages the use of physical objects like pen, water can, magic wand, wrist watch etc. to help people feel and sense real world objects to understand virtual interactions better. This kind of mimicking using physical objects (or paper + IoT) in the real world offers users a digital experience that is more analog in interaction.
2D interfaces lack depth and have boundary limitations in terms of experience because there is just a single point of contact with the membrane, as against numerous floating membranes in 3D, that separates the user from the interface. Hence, transferring interaction from 2D space to 3D is a not a pleasurable experience. As stated above, paper models can be converted into 3D architectural models to get a closer sense of what the 3D world looks like. Designing in three dimensions can be very challenging, but with the right tools and creativity, one can move beyond flat thinking.
While low-fidelity prototyping and VR prototyping exist and have just proved satisficing, a lot of these prototypes need to be tested and iterated upon because there is a chance of these being misrepresented and misunderstood by users, just because they are not real. There is a demand for technical standards that must be established, and better prototyping tools that must be stronger and easily accessible by designers and developers.
FUTURE OF VR
Virtual reality is directly tapping at the way we are perceiving and seeing the world. In the near future, people expect to see VR seamlessly integrated into lives to enhance and enrich everyday experiences by allowing to capture and relive best experiences with a whole new lot of possibilities. Virtual Reality will get more physical that will allow users to participate physically in the virtual world. With Facebook adopting VR to enable better social connections with Oculus, virtual reality is going to be big in games and social media, enabling people to experience the impossible. Virtual education tours can be used to make experiences more meaningful and memorable for students. Visualization can be used as an effective collaborative tool in design review processes to engage with stakeholders and to obtain feedback. VR sees a huge demand in journalism, automobiles, aviation, architecture etc. with teleportation and better prototyping made possible.
Beyond just games, other forms of entertainment would also be benefited from VR technology. Theater audiences can enjoy 3D movies with apps like Oculus and be more immersed in their movie experiences. VR can transport users to an imaginary concert performance shot by virtual reality filming. The tourism industry can leverage the VR technology to satisfy people’s wanderlust. Healthcare has been a big adopter of virtual reality tech. Virtual models would help surgeons determine the safest and most efficient way to locate tumors, place surgical incisions or practice difficult procedures ahead of time. Also, VR can be used as a cost-effective and engaging tool for rehabilitation. Users can put on their Oculus headset and teleport to find their happy place and be immersed in a relaxing environment. VR can be used as a therapeutic tool to reduce stress and anxiety from daily life. Space scientists can use VR to control robots on different planets and provide a way for astronauts to de-stress. Museums can adopt VR tech to enable art connoisseurs to explore and experience the museum’s physical collections. In the fields of education, virtual reality can make cognitive learning faster, more effective and efficient. The military can explore using VR game-like simulations as collaborative tools to practice and experience virtual environments before using real-world tactical equipment. Courtrooms can use VR to recreate or reconstruct crime scenes and help the jury to understand and visualize the details of the case and make decisions accordingly.
Creating delightful experiences in VR that are closer to how humans interact in the natural world is challenging and demands better technical standards. As much as you can create amazing experiences, you can also fall into the pit with creating bad experiences. A very faint line separates the two and hence there is a need for more qualitative research in this field. Virtual reality has raised the bars and work here must meet expectations set by users. Graphics and sound have gotten great in virtual reality with the release of Oculus Rift and HTC Vive. But for all the progress that virtual reality is making, we haven’t discovered methods to best control and play with objects that we find on the screen. People are looking for the real feel and there is a hunger for genuine virtual reality experiences.
UX/ UI designers are the architects of the world that is yet to come. They must not write off VR tech as a passing craze but must instead leverage upon the ‘out of the box’ thinking it offers for design and develop better design ideas. By working with designers and developers across the industry, it is possible to make this kind of reality part of daily life for billions of people. Virtual reality was once a dream of science fiction, so were the internet, computers and smartphones. With better design and better interactions, virtual reality is sure to unlock new worlds for people.
Vamien McKalin. 2014. Augmented Reality vs. Virtual Reality: What are the differences and similarities? Tech Times. http://www.techtimes.com/articles/5078/20140406/augmented-reality-vs-virtual-reality-what-are-the-differences-and-similarities.htm
Jonathan Ravasz. 2016. Design practices in virtual reality. uxdesign.cc — User Experience Design. https://uxdesign.cc/design-practices-in-virtual-reality-f900f5935826#.r453jp7ou
2016. Buttons in virtual reality — a UI/ UX design approach. Reality Shift. http://realityshift.io/blog/buttons-in-virtual-reality-a-ui-ux-design-approach
Virtual Reality and Interaction. Carnegie Mellon University. http://graphics.cs.cmu.edu/nsp/course/15-462/Spring04/slides/23_vr.pdf
Jonathan Strickland. 2016. How Virtual Reality works. How Stuff Works Tech. http://electronics.howstuffworks.com/gadgets/other-gadgets/virtual-reality3.htm
Instr. Gökhan Nalbant, Instr. Barbaros Bostan. Interaction in Virtual Reality. http://newmedia.yeditepe.edu.tr/pdfs/isimd_06/12.pdf
Heim, M., “Virtual Realism”, Oxford University Press, 1998, p.7
Rachel Metz. 2016. The Step needed to make Virtual Reality more real. MIT Technology Review. https://www.technologyreview.com/s/543316/the-step-needed-to-make-virtual-reality-more-real/
Sam Ashcroft. 2015. Interaction is key to a decent real world VR experience. Wareable. https://www.wareable.com/vr/interaction-is-key-to-a-decent-vr-experience-1493
2016. The Best Way to interact with VR. Upload. http://uploadvr.com/the-best-way-to-interact-within-vr/
Sara Breselor. 2016. How we did it: Prototyping in Virtual Reality. Ideo Labs. https://labs.ideo.com/2016/03/07/how-we-did-it-prototyping-in-virtual-reality/
Amir Khella. 2016. How to prototype for Virtual reality in Virtual reality. A Medium Corporation. https://medium.com/@amirkhella/how-to-prototype-for-virtual-reality-in-virtual-reality-af78336bd947#.njvw57f1x.
Kent Vasko. 2016. Design Prototyping in Virtual Reality. Maya. http://maya.com/blog/design-prototyping-in-virtual-reality
Nicolas Barrial. 2015. How to prototype in virtual reality. Makery media for labs. http://www.makery.info/en/2015/06/23/prototyper-en-realite-virtuelle-modes-demploi/
Timmevandermeer. 2016. Prototyping interactive environments in virtual reality with Google Cardboard, Unity and Hotline Bling (TfCD). Instructables. http://www.instructables.com/id/Prototyping-Interactive-Environments-in-Virtual-Re/
Jody Medich. 2015. Break out your scissors: The secret of rapid 3D prototyping for AR/VR and IoT. LeapMotion. http://blog.leapmotion.com/break-scissors-secret-rapid-3d-prototyping-arvr-iot/
Rob Spiegel. 2016. Virtual Reality: The New Prototype is Digital. Design News. http://www.designnews.com/author.asp?section_id=1386&doc_id=280554
Bill Howard. 2014. Inside Ford’s virtual reality prototyping lab: Why make full-size clay cars when VR will do? Extreme Tech. http://www.extremetech.com/extreme/185109-inside-fords-virtual-reality-prototyping-lab-why-make-full-size-clay-cars-when-vr-will-do
Nick Kizirnis. 2015. Why we love rapid prototyping for Virtual Reality apps. Marxent @ work. http://www.marxentlabs.com/job/why-we-love-rapid-prototyping-for-virtual-reality-apps-marxent-work/
Ilan Regenbaum. 2016. The bright future of virtual reality. TNW. http://thenextweb.com/insider/2016/03/07/the-bright-future-of-virtual-reality/
Knvul Sheikh. 2016. Beyond Gaming: 10 Other Fascinating Uses for Virtual-Reality Tech. Live Science. http://www.livescience.com/53392-virtual-reality-tech-uses-beyond-gaming.html