AR Visuals in Educational Books

Shreyas Agnihotri
Oct 24, 2019 · 5 min read

Expanding the capabilities of paper as an information carrier: a case study


As the second project in my Human-Computer Interaction class at HKUST, I was tasked with developing an Augmented Reality application to improve on the information-sharing capabilities of paper. The ultimate application would use mobile device to scan paper and demonstrate an interactive or 3D model. The context of the application (education, health care, etc.) and type of paper product (books, sticky notes, etc.) remained up to us. Below is a broad documentation of our process and my reflections:

Initial Concepts

An initial mind-map brainstorming categories of paper products, possible user demographics, and value propositions

My group’s initial brainstorming session brought us to 2 key areas: textbooks and magazines. Each had distinct usage scenarios, in education and shopping respectively, and very different potential value-adds to the paper experience. Our preliminary interviews with users hinted that educational applications were the most compelling, given students’ receptiveness to interactive learning and new technology. With these categories in mind, we developed a few concrete AR ideas:

  • Foreign languages: Improving foreign language learning through word visualizations.
  • Textbooks: Providing live concept animations and educational tutorials within textbooks.
  • Children’s books: Adding delightful animations and visuals to children’s books.
  • Product catalogues: Allowing shoppers to preview an item from a magazine or catalogue within their environment, or even try it on.

Each of these resulted in a storyboard like the one below.

Sample of an AR app that scans and displays animals on the phone to aid in children’s understanding of visuals and scale.

User Feedback

Bringing our storyboards to potential users, we got surprisingly strong feedback in favor of one of the ideas: textbook visuals.

“I would have loved some interactive games and visuals in textbooks growing up. It would have broken up the boring reading.”

Our other initial ideas, such as catalogue visuals and language-learning aids, received much more lukewarm reactions.

“I don’t really think I’d want to preview my clothes on an app when I can just go into a store to try them. Plus, I already know my size and the pictures online give me a sense of the product.”

“There are already really good solutions for language-learning, like Duolingo, so I don’t know how useful it would be to have a separate app to visualize foreign words.”

Almost all of our users related to the feeling of wanting more from their textbooks, and felt they’d learn better and faster with the help of an engaging mobile tool. In contrast, the product preview and language-learning applications felt like unnecessary additions to existing products.


Apple’s RealityKit development environment for iOS.

Our prototype centered around one specific application: visualizing animals for use in children’s environmental science and biology books. Rather than learning new animals and body parts on a 2D plane, students can scan a word or code on the page to bring the visualization to life.

How ARKit and RealityKit interact to create this application.

The end product uses Apple’s RealityKit to detect 2D planes and user interactions. Behind the scene, ARKit detects horizontal planes to construct the real world, while RealityKit combines the real world and virtual world. RealityKit can then alter objects in the reality composer and trigger behaviors that we define.

In this case, animals float above the page, and a touch from the user causes them to move to the forefront and display a label.

Video Demo

An example of how to use the AR application and the user’s perspective

Contribution & Reflections

My main contribution to this project was in ideation and user testing. While brainstorming initial ideas, I led the team’s discussion of potential educational applications and conceptualized the storyboard for each. Once we had a prototype in hand, I also took the lead in bringing it to users through speed-dating, while collecting data and feedback. I engaged in user interviews with multiple friends and randomly-selected participants from the library, and developed a question-based framework to identify which areas of each product to improve. It was through these questions that we identified which products end users were actually interested in using.

In the process of this design sprint, I had a few key realizations about AR technology:

  1. AR development is hard. While working on the physical prototype, my group ran into multiple roadblocks that limited the potential of our demo. Unity, the main tool used in class, seems to have a steep learning curve when it comes to complicated interactions like rotations and animated behavior. Our second option, RealityKit, offers a more plug-and-play option, but still required us to convert our 3D files into alternative formats in order to make them work. The development of AR apps will hopefully only get easier as more applications of the technology start to emerge.
  2. AR technology is useful only in certain, specific cases. AR is an extremely exciting concept, and feels like the futuristic tech we expect from sci-fi movies. The reality, though, is that the act of mixing the digital and physical worlds through a screen often complicates tasks instead of simplifying them. Most of our initial storyboards presented novel ideas, but they turned out to be slower or less informative than existing solutions (like just going to the store to shop for clothes). It’s important that we don’t equate new with better as more technology like this emerges.

I appreciated the help of my fantastic group-mates during my first experience with AR design, and hope to continue exploring its potential applications within HCI in the future.

The Startup

Get smarter at building your thing. Join The Startup’s +788K followers.

Sign up for Top 10 Stories

By The Startup

Get smarter at building your thing. Subscribe to receive The Startup's top 10 most read stories — delivered straight into your inbox, once a week. Take a look.

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

Shreyas Agnihotri

Written by

The Startup

Get smarter at building your thing. Follow to join The Startup’s +8 million monthly readers & +788K followers.

Shreyas Agnihotri

Written by

The Startup

Get smarter at building your thing. Follow to join The Startup’s +8 million monthly readers & +788K followers.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store