As the second project in my Human-Computer Interaction class at HKUST, I was tasked with developing an Augmented Reality application to improve on the information-sharing capabilities of paper. The ultimate application would use mobile device to scan paper and demonstrate an interactive or 3D model. The context of the application (education, health care, etc.) and type of paper product (books, sticky notes, etc.) remained up to us. Below is a broad documentation of our process and my reflections:
My group’s initial brainstorming session brought us to 2 key areas: textbooks and magazines. Each had distinct usage scenarios, in education and shopping respectively, and very different potential value-adds to the paper experience. Our preliminary interviews with users hinted that educational applications were the most compelling, given students’ receptiveness to interactive learning and new technology. With these categories in mind, we developed a few concrete AR ideas:
- Foreign languages: Improving foreign language learning through word visualizations.
- Textbooks: Providing live concept animations and educational tutorials within textbooks.
- Children’s books: Adding delightful animations and visuals to children’s books.
- Product catalogues: Allowing shoppers to preview an item from a magazine or catalogue within their environment, or even try it on.
Each of these resulted in a storyboard like the one below.
Bringing our storyboards to potential users, we got surprisingly strong feedback in favor of one of the ideas: textbook visuals.
“I would have loved some interactive games and visuals in textbooks growing up. It would have broken up the boring reading.”
Our other initial ideas, such as catalogue visuals and language-learning aids, received much more lukewarm reactions.
“I don’t really think I’d want to preview my clothes on an app when I can just go into a store to try them. Plus, I already know my size and the pictures online give me a sense of the product.”
“There are already really good solutions for language-learning, like Duolingo, so I don’t know how useful it would be to have a separate app to visualize foreign words.”
Almost all of our users related to the feeling of wanting more from their textbooks, and felt they’d learn better and faster with the help of an engaging mobile tool. In contrast, the product preview and language-learning applications felt like unnecessary additions to existing products.
Our prototype centered around one specific application: visualizing animals for use in children’s environmental science and biology books. Rather than learning new animals and body parts on a 2D plane, students can scan a word or code on the page to bring the visualization to life.
The end product uses Apple’s RealityKit to detect 2D planes and user interactions. Behind the scene, ARKit detects horizontal planes to construct the real world, while RealityKit combines the real world and virtual world. RealityKit can then alter objects in the reality composer and trigger behaviors that we define.
In this case, animals float above the page, and a touch from the user causes them to move to the forefront and display a label.
Contribution & Reflections
My main contribution to this project was in ideation and user testing. While brainstorming initial ideas, I led the team’s discussion of potential educational applications and conceptualized the storyboard for each. Once we had a prototype in hand, I also took the lead in bringing it to users through speed-dating, while collecting data and feedback. I engaged in user interviews with multiple friends and randomly-selected participants from the library, and developed a question-based framework to identify which areas of each product to improve. It was through these questions that we identified which products end users were actually interested in using.
In the process of this design sprint, I had a few key realizations about AR technology:
- AR development is hard. While working on the physical prototype, my group ran into multiple roadblocks that limited the potential of our demo. Unity, the main tool used in class, seems to have a steep learning curve when it comes to complicated interactions like rotations and animated behavior. Our second option, RealityKit, offers a more plug-and-play option, but still required us to convert our 3D files into alternative formats in order to make them work. The development of AR apps will hopefully only get easier as more applications of the technology start to emerge.
- AR technology is useful only in certain, specific cases. AR is an extremely exciting concept, and feels like the futuristic tech we expect from sci-fi movies. The reality, though, is that the act of mixing the digital and physical worlds through a screen often complicates tasks instead of simplifying them. Most of our initial storyboards presented novel ideas, but they turned out to be slower or less informative than existing solutions (like just going to the store to shop for clothes). It’s important that we don’t equate new with better as more technology like this emerges.
I appreciated the help of my fantastic group-mates during my first experience with AR design, and hope to continue exploring its potential applications within HCI in the future.