SignVision — A Real-time Mobile Sign Language Recognition Application built using ChatGPT
A Real-Time Sign Language App built by two Nanyang Polytechnic Students
- An idea that came out of nowhere. From ideation to pitching a Real-Time Sign Language Translator to the world.
- Pushing their limits, two best friends figure out how to interpret Sign Language using AI
- This is a reflection of our 3-month journey to develop our biggest project yet, SignVision.
The idea came about when, in early 2023, Iannuzzy and Nasrullah were hanging out at Nasrullah’s house, thinking about the next idea that they could launch. What they both knew from the start was that it had to be linked to something new, something emerging. AI was definitely the angle Iannuzzy and Nasrullah were going for, but they needed a good topic. All it took was one simple YouTube video of a simple Python program that reads signs based on stored data.
It clicked on both of them. Both boys have relatives who suffer from hard of speaking and hearing, leading to communication through “homemade” sign language. Current sign language translation can be done either by learning sign language yourself or getting a translator to translate for you. What if one can translate sign language instantly without the need to go through the process of learning or needing third-party services to do it for them? That is the start of SignVision, a real-time sign language translator that allows anyone to translate sign language on their own at any point, anywhere.
Nasrullah, the main driver behind building AI and machine learning models, took charge of learning all the skills he needed to build the application and practise new coding languages and models to build such a prototype. Iannuzzy took charge of the market research and the possible tools to make this product a reality. It came as a genuine shock when both of them found out that their very school was taking on applications to take part in the Lee Hsien Loong Smart Nation Award. This award differs from any other; it is a national award for only a few selected students. Iannuzzy and Nasrullah had to take part in it.
SignVision held the potential to clinch the Lee Hsien Loong Smart Nation Award, but its success hinged on proving its viability. The concept of SignVision was promising, yet several critical questions needed answers to refine the product further:
- Design and Interface: What will the product’s physical and user interface look like? It’s essential to consider how users will interact with SignVision in various environments.
- Technological Requirements: What specific technologies are required to enable real-time translation of sign language? This involves identifying the necessary software, sensors, and perhaps machine learning algorithms.
- Market Fit: Does SignVision effectively address a genuine need within the target market? Understanding the demand and how the product fits into existing communication practices is crucial.
- Usability: Can anyone use it effortlessly? Ensuring that the product is user-friendly for people of all ages.
- Product Form: Is SignVision a standalone hardware device, or is it software that can be integrated into existing devices like smartphones or tablets?
The journey to perfect SignVision involved numerous long nights as Iannuzzy and Nasrullah delved into the intricacies of their innovative product. Through relentless dedication and iterative improvements, each version of SignVision surpassed the last. After two months of rigorous development, the final product emerged as a sleek web application. This platform facilitates a frictionless onboarding process, allowing users to quickly engage with the tool. By simply visiting a URL, users gain instant access to on-demand sign language translation, making communication more accessible than ever before. This web app model not only streamlined the user experience but also significantly broadened the potential user base by eliminating the need for specialized hardware.
The web application of SignVision harnesses the power of AI and deep learning to interpret sign language gestures and convert them into spoken English. Initially, the MVP (Minimum Viable Product) version of the app is capable of translating approximately 50 signs from Singaporean Sign Language, utilizing a comprehensive data library from the Singapore Sign Language Sign Bank. To ensure the translations are not only accurate but also contextually appropriate, Nasrullah incorporated a critical feature using OpenAI’s ChatGPT API. This integration enables the app to reformat basic sign expressions — such as the two-word phrase “Toilet, where” — into more coherent and understandable sentences like “Where is the toilet?” This sophisticated linguistic transformation significantly enhances the usability of SignVision, making it a more effective communication aid.
There are, of course, still limitations to this web app. Singaporean Sign Language needs to be completed, and Singaporean signs have scarcely been updated to add them. This makes the application hard to translate when there is a limited sign language “dictionary” available. The current workaround would be to use ASL (American-Sign-Language) to compensate for the lack of words in the Singaporean Sign Language. The web app also relies on the camera quality to read signs to be able to translate them, if a room is too dark or the web app is not trained to understand the slight difference in hand gestures. This also applies to varying skin tones, due to a lack of data on different people signing sign language, creating a vulnerability to data biases.
SignVision is intended to be a transformative tool designed to help those who do not understand sign language communicate more easily with the deaf, yet there is ample room for further enhancement to maximize its effectiveness. Key areas for development include:
- Optimizing Memory Usage: Implementing more efficient algorithms that put minimal stress on system memory, ensuring smoother operation across a variety of devices.
- Expanding Training Data: By training the AI on a larger dataset gathered from a diverse audience, the model’s accuracy and robustness can be significantly improved. This would allow for a more inclusive range of sign language interpretations.
- Supporting Longer Phrases: Enhancing the model to handle long and complex sign language phrases will make the tool more practical and versatile in everyday communications.
- Global Scaling: Developing models that recognize sign languages from different parts of the world will turn SignVision into a globally applicable tool, breaking communication barriers on an international scale.
- Community-Driven Updates: Plans are in place to open the web app’s library to the public, allowing users to contribute their own sign language inputs. This community-driven approach can vastly expand the app’s sign language database and improve its adaptability.
- User Interface Enhancements: The team also aims to refine the web app’s user interface to enhance usability. Creating a guidebook is one proposed strategy to help new users navigate the app more effectively, providing clear instructions and tips to get the most out of SignVision.
Like-minded individuals and organisations took an interest. One such company known as GUILD (Ground Ups Innovation Labs for Development) took great interest in the potential of SignVision. Ibnur Rashad, Chief Foresight Officer for GUILD, and Nazeera Mohammed who is a Community Leader and Volunteer collaborated and helped Iannuzzy and Nasrullah by providing user feedback and insights into how the web app can be implemented.
SignVision was a product made from sheer hard work and a consistent will to keep trying again and again. The product won its place in the competition, and both of them won the Lee Hsien Loong Smart Nation Award in May 2023.
This award was truly a milestone that overshadowed their previous challenges and setbacks with satisfaction and pride for their work. It was truly an honour to walk up on stage to receive the award from the Minister of State, Ministry of Education & Ministry of Manpower, Ms Gan Siow Huang. Being surrounded by other awardees also allowed both Iannuzzy and Nasrullah to sink in the fact that their hard work and dedication are shown and that they earned their place here.
The support from their families and mentors, who encouraged both boys to pursue this project, kept both of them on the right track and helped them take the next steps to bridge the communication gap between us and the deaf.
SignVision’s impact and innovation caught the attention of those at Block 71, a prominent tech-startup ecosystem incubator, leading to an invitation for Iannuzzy and Nasrullah to speak about their project. SignVision is a defining moment for both Iannuzzy and Nasrullah, and this project has gone a long way from being ideated from out of nowhere at home by two polytechnic students who are looking for a project. This is definitely not the end of SignVision, but it is still the beginning of a possible future in which communication barriers between us and those who are deaf can be torn down. This is also the first few steps of two best friends whose ambitions have yet to slow down.
*This publication was written collectively by Nasrullah and Iannuzzy.
