Exploring Mixed Reality Design Issues In Studio
Ralf Schneider, Syracuse University
Virtual Reality (VR), Augmented Reality (AR) and Mixed Reality (MR) are rapidly gaining popularity. The associated augmentation technology is improving at a fast pace and presents significant new interaction design challenges.
During the fall semester 2017, 13 students of the industrial and interaction design program at Syracuse University explored and identified design opportunities around specific themes in which MR could be a game changer. The students developed stakeholder-centered MR scenarios that, compared to the traditional non-MR experience, show an improved task-flow and deliver a compelling argument as to why mixed reality can improve certain types of experiences and outcomes. This semester long project focused on investigating mixed reality applications considering the potential of the head mounted display (HMD) technology.
The students had access to one Microsoft HoloLens, a state of the art HMD device. This circumstance is similar to studying Industrial Design in the 1990s, when for example students were competing for one SGI workstation to learn Alias/Wavefront Studio surface modeling. To compensate for this limitation, the device was available for checkout during the week. The portability of the HoloLens simplified this process.
Rich Hanks, principal designer at Tactile, agreed to support the project as an external expert. He was introduced at the beginning of the semester via Skype and joined the midterm review in person. Students valued the input from designers working in the field.
In order to focus the inquiry, students could choose from three specific areas of exploration:
1. How could mixed reality impact machinery solutions for industrial process automation and integration?
(Primary stakeholders: engineers & technicians)
Bartell, a company that manufactures machines for the tire industry, was the connection for this theme. This engagement allowed the students to gain a better understanding of various stakeholders. Furthermore, it strengthens relations with the community.
2. How could mixed reality improve the fuzzy front end of the design process?
(Primary stakeholders: designers & design managers)
The students decided to create solutions for fashion designers.
3. How could mixed reality benefit the stakeholders of a Fire Department?
(Primary stakeholders: fire fighters)
The learning curve to test a MR idea on the HoloLens is steep. Programing skills and a fluent use of Unity is required to create an app and install the program on the HoloLens. In a 15-week semester, this could be an obstacle for design students, depending on the students’ background. As an alternative, the students experimented with Thingworx, an MR authoring software that lowers the barrier since programming skills are not needed.
The required reading for the course was the publication 3D user interfaces. This book provided many deep insights on the subject matter. Especially the design guidelines helped the students to improve their design solutions.
Semester Journey
In an MR environment, the confluence of reality and digitally enhanced reality needs to be carefully modulated. The blending process requires constant questioning on how much or how little information is displayed in the user’s field of vision and the level of interactivity. This issue was explored at the beginning of the semester when the students were tasked with developing a MR concept for a prospective student visiting campus. An analog ideation design sprint was followed by a more refined sprint, which included creating photoshopped images for storyboards. Finally, the students crafted 45 second videos which required them to develop a coherent story of the experience. Without any programming skills and without the use of AR technology, but with an understanding of MR possibilities, students communicated their ideas while playfully immersing themselves in the topic.
At the same time, students developed a deep understanding of their chosen area of exploration. In groups of five, they talked to the user base to understand stakeholder’s needs. They visited the local fire station, went on a fieldtrip to Bartell and interviewed fashion design students. Students immersed themselves in the field of mixed reality and learned about issues of use associated with the HoloLens through benchmarking current HoloLens apps (HoloMaps, Holotour, Roboraid, etc.). A research report gathered the takeaways.
For concept ideation, the students worked in groups of two. The MR development process was guided by the question: “How can we simulate a digitally enhanced, immersive experience to persuade a potential client to commit to production?” This senior level design studio exposed students to systems thinking and pushed the development of a certain set of design skills. In addition to the thorough design research process needed to develop empathy for the user, UX/UI considerations for MR require the ability to think in 3D and to be able to ideate at the intersection of 3D interaction and time. The traditional process of wireframing a 2D interaction proved to be insufficient.
To overcome this challenge, the student teams hinged their projects on scenario-based ideation, which drove the use of analog and digital tools. The students created low fidelity prototypes using clear acrylic on which they sketched possible MR content and sketched on photosphere templates to visualize in 3D on a 2D document. They also utilized MR interaction sequence templates, which call out the possible interaction.
Rich Hanks, the external expert on the subject matter, visited the studio for the midterm review. This added subtle pressure in bringing the students’ ideas to a presentable level. Each team received feedback to refine the work. The review surfaced that the students were holding on to the 2D paradigm, familiar from the use of smart phones and tablets. They were encouraged to explore the possibilities of 3D interaction.
While refining the projects for the final presentation, the students excelled at communicating their ideas with comprehensive visualizations, animated gifs and videos. Below are selected project examples of the envisioned mixed reality experience.
Conclusion
1. Critical inquiry in all phases
It is important to ask critical questions throughout the design process, due to a lack of comparable examples and the need to justify why an augmented solution is better than the non-augmented solution.
2. Exploring 3D interaction possibilities is hard work
Students are used to the 2D paradigm of smartphones. Encourage students to explore 3D Interaction possibilities from day 1.
3. 3D Information graphics
Information can be displayed in 3D and interconnected with big data.
4. Sketching
To accelerate the iterative design process, sketching (analog/digital) remains valuable.
5. Visual Narrative
Tell the story in a sequence with compelling images that highlight the advantages of the new idea. The best student outcome included a short video.
An academic setting allows students to experiment in emerging technology spaces — honing skills that could lead to promising career opportunities. Design educators should take an active role in shaping the potential of mixed reality interactions, where reality and augmented reality flow together.
About Ralf
Ralf Schneider is an assistant professor of industrial and interaction design in the School of Design, College of Visual and Performing Arts at Syracuse University. As a designer and design educator, he combines European and American design values. In addition to teaching industrial design at Wentworth Institute of Technology in Boston and DAAP in Cincinnati, Schneider worked as an associate director/senior design researcher at the Live Well Collaborative. Schneider is interested in solving complex problems with interdisciplinary teams. His current research focuses on design technology especially how mixed reality impacts design.
