Interaction Design Studio I — Project 3

Progressive User Interface — Augmented Reality Learning

Published in

MDES Process Work — Carnegie Mellon University

6 min readDec 9, 2015

Introduction

Our team was asked how user interface would look like in an immersive environment where digital information is seamlessly integrated into physical surrounding. With new technologies emerging in virtual and augmented spaces, such as Microsoft Hololens, Google Glass, and Occulus Rift, there is significant potential for design to help shape the future of these technologies.

Background

We choose to explore what learning could be in an augmented reality. We believe that augmented reality introduce a specific value proposition in bringing new aspects of physicality across distance and therefore decided to explore how that technology could facilitate the learning of physical skills.

Whether we’re learning how to use a tool, do a craft project, or play an instrument, augmented reality provides an opportunity to transmit new information across distances.

Concept

Bodhii is a learning application for a seamless real-time shared physical learning space. The key design goal of Bodhii is to allow a pair of users to transmit information needed to learn a physical skill by translating haptic feedback into visual information. At the same time, social, visual, and auditory learning aids provide an interactive and engaging learning medium.

User Interface

The interface is a combination of a 3D shared holographic space with supportive learning aids.

We imagine a application where:

two people can provide physical positioning information in a shared 3D space
the shared space transmits sounds seamlessly between the two spaces
natural language processing is used to activate additional features, such as haptic feedback which can be captured and represented visually
natural language processing can also be used to activate:

visual aids which can be incorporated into the learning experience
integrated video feeds which bring conversations between the pair to life
and other audio aids as needed

In addition, Bodhii is a learning application that is designed to help learners move through the different levels of mastering a skill.

Therefore, sessions and parts of sessions can be recorded to provide on-going teaching instruction in asynchronous formats for students and teachers. And as students move towards mastery, additional collaboration and performance features can be integrated into the platform for social sharing.

Resolving Issues and Areas for Continued Exploration

A couple of key issues we resolved through the process of discussing and prototyping were:

Device Manipulation

Height differentiation: If the student places the device on top of a table, and the teacher places it directly on the floor, the teacher may see the hologram at an uncomfortable height to interact with. For ease of storytelling, we decided to give them equal height for now, but any of the two parties would be able to adjust the height by lifting up/down the hologram with their hands, á la Iron Man’s lab in this video: https://www.youtube.com/watch?v=Wx7RCJvoCMc
Simplify buttons & defining radius: We’ve decided on a simple, circular device that is portable. Since the device doesn’t need too many physical features, it can be turned on/off by a swiping gesture on the side. Once it is on, you can keep twisting on the side of the device to adjust the radius of what the device should read.

2D vs. 3D: The body is visualized in a 3D fashion while the face and other visual learning aids are 2D.
Haptic “input” / visual feedback: Both the teacher and the student can view haptic feedbacks from the other visually. Upon cue, the student’s pressure points on the object would get displayed from the teacher’s POV, and when the teacher interacts with the hologram of the student’s hand and pushes downward in the vision (i.e. to “exert” pressure), the student sees the teacher’s hologram-hand trying to exert pressure on her hand, via the color code. Once the technology exists for haptic feedback transmission over distance, we would incorporate this technology into the application as well.

Scenario

Here is a sample of what it one scenario might look like with the application where a student is learning to play an instrument:

1-to-1 Live Prototype

Brainstorming & Process

We started with a brainstorming session. We realized from our first projects that a common theme through each of our proposed applications was the underlying purpose to communicate and educate as mediated through technology.

After identifying the area of learning to focus on, we approached the idea from a few different angles:

What would it look like? What kinds of features could the learning application have? (e.g. holographic projection, two-way interaction?)
What was the scope of the application? What types of learning could be done on the platform? (e.g. one-to-one vs one-to-many, synchronous vs. asynchronous?)
What could it be used for? What specific things could people use the application to learn? (e.g. dancing, cooking, playing instruments?)
Information. What types of information/data would be transmitted/used in the application? (e.g. physical through visual and haptik, smell?)

Brainstorming concept ideas and scope

Initial visualizations of the application

STORYBOARD

SETUP

The setup of pepper’s ghost became a key aspect of our project. We researched ways to create a one-to-one in-person holographic effect by using translucent materials (e.g. glass, plexiglass) and a screen (e.g. laptop screen, TV screen, and projector). One of the issues with using this effect is that pepper’s ghost produces 2D effects. However, our application proposes a 3D space with 2D components.