SyncItUp: Designing AR Glasses to Empower Cross-Lingual Communication

Introduction

In a world increasingly dominated by technological advances, we often overlook a fundamental challenge faced by millions: the communication gap for the Deaf and Hard of Hearing (HoH) community. Our project, SyncItUp, aims to bridge this gap using Augmented Reality (AR) glasses, transforming cross-lingual interactions and allowing for more natural, hands-free communication. This article dives into our design process, challenges, and the insights we gained along the way.

Project Overview

SyncItUp was born out of a passion project by three graduate students at Pratt Institute, as part of our Extended Reality course. With a tight timeline of 1.5 months, our interdisciplinary team — two designers and one video editor — collaborated to bring this concept to life, relying on tools like Lens Studio, Figma, FigJam, Blender, and Adobe After Effects.

Challenge

Time and technical constraints were significant hurdles. Within a short 1.5-month timeframe, we needed to research, ideate, and develop a functional prototype to demonstrate our vision. We were determined to create an assistive product that could minimize the need for external aids, focusing on face-to-face conversations between Deaf and hearing individuals.

Our Goal

In the era of wearable displays, we aimed to design an intuitive cross-lingual communication interface, synchronizing communication between Deaf and hearing individuals while considering the diversity in sign languages and communication needs.

Problem Discovery

Understanding & empathising with the deaf community

To build empathy, we interviewed Deaf individuals and watched video interviews. They shared insights that greatly influenced our design:

“Sign languages are not universal. They are just as diverse as spoken languages.”

“We are a linguistic minority with a unique language and culture.”

Assistive Resources in the Market

Current assistive solutions include real-time captioning apps, translation apps, and ASL learning platforms. While they help, most require external devices like smartphones, which interrupt natural interactions. SyncItUp’s AR glasses aim to eliminate this dependency by integrating essential features into the wearable itself.

To find more tools and resources, refer to this comprehensive list made by Toby Flitch 💜

Defining the Problem

The core problem was the communication gap between Deaf and hearing individuals, exacerbated by a lack of accessible resources like certified interpreters and sign language training.

Who is SyncItUp for?

SyncItUp is designed for Deaf and Hard of Hearing individuals seeking a hands-free, real-time communication aid. While the primary user group is Deaf individuals, the glasses also assist hearing individuals wishing to learn and communicate in sign language.

Choosing the Device

Given the project’s focus on Extended Reality, we decided to use AR glasses. This decision stemmed from our desire to make communication as natural as possible, allowing users to engage hands-free without looking away to another screen.

Ideation & Implementation

SyncItUp’s features were divided into three primary modes:

Communication Mode
This core mode facilitates two-way interpretation using AR. The glasses detect hand movements and gestures to translate sign language into spoken language for hearing users, and vice versa, providing real-time AR captions for Deaf users.

• The glasses detect hand movements and gestures as the user signs
• The glasses translate sign language into spoken language using the in-built microphone for the hearing person
• The glasses recognize speech and convert it into text
• The glasses provide AR captions for the sign language speaker

[video]

Communication mode for emergency
SyncItUp also provides users with a feature for emergencies -

• Users can sign a gesture to activate the emergency mode
• The glasses will provide a list of 4 contacts for the user to choose from
• The glasses initiates the call after user’s confirmation

Customization Mode
Customization allows users to add new signs or jargon to their vocabulary, making SyncItUp more adaptable and personalized to their unique communication needs.

• The user can customize the glasses according to their vocabulary
• The user can add new signs to the sign language database for a word/ phrase/ jargon

[video]

Learning Mode
For those learning sign language, SyncItUp offers interactive lessons, with feedback on signing accuracy, tailored to different learning levels.

• The glasses offer different learning levels of any sign language
• The glasses teach new signs and phrases
• The glasses provide feedback on the learner’s signing accuracy

[video]

Learning mode for practice

• The learner can practice sign language with an augmented person
• The practice experience adapts to the user’s level of learning
• The user can choose to learn and practice simultaneously.

These modes were conceptualized through storyboarding and further refined through user flows and wireframes in Figma and FigJam.

Testing

I engaged three potential users — a Deaf individual, and two interpreters— to test the concept. Feedback underscored the importance of tailoring the device for varying levels of hearing sensitivity and understanding user comfort.

Learnings

Extensive primary research was essential. Every user we spoke to emphasized different needs, leading us to realize that this technology’s utility would vary widely among individuals. We also saw the potential for SyncItUp to become an invaluable tool in specific contexts, even if it wasn’t a universal solution.

Next Steps

To build on our concept, we identified several next steps:

• Conduct deeper primary research with Deaf individuals, interpreters, and accessibility experts.
• Expand the vocabulary and language support.
• Explore partnerships with AR hardware developers for seamless deployment.

Reflections

Working on SyncItUp deepened my appreciation for sign language and the Deaf community. This project not only motivated me to continue learning American Sign Language but also highlighted the value of involving users directly in the design process. SyncItUp is only the beginning; I’m committed to contributing to accessible design that brings us closer to a world where communication barriers no longer exist.

Conclusion

SyncItUp represents more than just a product; it’s a step toward inclusivity. AR technology has the potential to create more accessible, natural communication experiences. While our project is still a prototype, the journey has solidified our commitment to designing for social impact and accessibility.

Thank you for reading! If you’re passionate about inclusive design, I’d love to connect! Reach out to me at nehalsharma1703@gmail.com, or find me on LinkedIn.