A11y - Super-vision for those who need it
On one of my numerous grocery runs, I noticed a man who had lost his sight, begging by the entrance of the neighborhood supermarket. I felt sorry for him and approached to give him some money. His diction was amazing so I engaged him in a conversation. Apparently, he had lost his sight the previous year in a gruesome accident that claimed the lives of his entire family and due to his inability to effectively carry out his job effectively, he was laid off by his company shortly after. This got me thinking, what if I could create a digital eye for the visually impaired, an app that would improve the quality of their lives and serve as an artificially intelligent companion?
My curiousity led to the creation of A11y. A11y (Accessibility) is an innovative and simple-to-use gesture-based intelligent camera assistant that aids blind and/or auditory impaired people in performing tasks in their everyday lives.
Side note: It’s called A11y because there are 11 letteres between A and Y in Accessibility, get it? ;)
Research
To adequately capture the issues faced by this particular category of users, I had to conduct interviews. Thankfully, my time serving as a volunteer at Be My Eyes exposed me to a wide range of limited vision individuals. I conducted interviews with 12 of these users, asking about the assistive tech they currently use in their day-to-day activities and the problems they encounter while using these tools. Their responses revealed the following pain points:
- performing specific actions on a screen with multiple options;
- the absence of non-visual representations where required;
- navigating through contextually irrelevant data on the screen.
These grievances birthed Theresa, the persona I would be using in this study. She is a millennial who was diagnosed with Leber’s Congenital Amaurosis when she was born.
Providing solutions with A11y
A11y has 3 base modes that provide solutions to some basic problems faced by Theresa. These include the general, text scanner, and assistant modes. Additionally, with the aid of the magnifying tool, Theresa may choose to zoom in and out of the identified scenery, sharpen edges, increase contrast, and change color combinations. She may also contact family and friends to help out in times of emergency or technological limitations with the aid of the Emergency call feature.
Identifying Sceneries and Objects with the General Mode
By default, the General mode scans environments and lets users know where she is at every point in time. By tapping and holding the camera icon, a wheel with four additional features is revealed. These features may be navigated by moving one’s fingers in a circular motion.
- Identifying Sceneries: A major issue that Theresa faces is identifying places, especially when she has never visited a particular location before. This feature acts as a scanner for the environment within the camera lenses. When it captures the particular scenery, Ally notifies users of what the environment looks like (e.g., a living room, restroom, or a restaurant).
- Shopping aid: With the shopping aid, Theresa can have a more seamless shopping experience. It helps in navigating through stores easily and identifying products that she intends to purchase. This may be achieved by simply pointing the camera lens at the particular product. Ally brings the product description and price to her attention.
- Identifying currencies: One of the major accessibility problems that visually impaired users face is identifying currencies. When a user is in this mode, Ally scrutinizes the currency note/notes and informs the user.
- Aiding hearing capabilities with the lip-reader feature: This feature was specifically designed for users with auditory impairments. Once in focus, the lip reader feature examines lip movement and translates this into text, which is read aloud to the particular user.
- Identifying objects: This feature enables users to identify particular objects in a scene and provides adequate descriptions of the objects. All the user has to do is point the camera to the object and a description of it is read aloud.
According to a study by the World Health Organization, at least “2.2 billion people” globally, have a near or distance vision impairment.
Optical Character Recognition
Optical character recognition (OCR) technology is a system for extracting data from printed or written text in a scanned document or image file and then transforming the text into a machine-readable format that a user can readily consume. The scanned text can be read aloud immediately in synthesized speech. This makes virtually any typed or printed material accessible to the blind user.
Implementation on A11y: Transcripts can be generated from documents obtained using the text scanner’s camera. To accomplish this, the camera takes an image of the written text. The converted text is then read out to the user, or in this case, Theresa. By tapping in the center of the screen, she can Pause and continue the transcription. Additionally, the transcript can be rewound by tapping and holding on the left side of the screen and fast-forwarded by doing the same on the right side
Finding objects and performing tasks with “Ally” under the Assistant Mode
The two features under the assistant mode are the Find or Search feature and the Task-Performing feature.
Under the find or search feature, Theresa may ask AllY (the virtual assistant) to find a certain object and then point the camera lens in different directions while Ally attempts to find it by scanning the premises.
Ally may also act as a conventional assistant when asked to perform basic tasks such as ordering some food, booking flight tickets, calling a contact, and can even make jokes among other things.
Contacting Help with the Emergency Caller
At the top right of every mode is a caller icon. By tapping on this, Theresa can call an already predetermined emergency contact through video call.
Other General Feature
- Magnifier and Contrast functions- At the top left of the screen is a magnifier that enables Theresa to zoom in and out of scenery by simply sliding up or down. Beside this icon is the contrast option, which she may use to increase or reduce the contrast of the particular environment on her screen. This could be done by also sliding up or down when the feature has been selected.
- Recommendations on what mode should be used for each case: In a scenario where Theresa is attempting to perform a task with the wrong mode, Ally makes recommendations on what mode will be best suited for the task.
Lessons Learned from this Project
- There is a need for technology to be more encompassing i.e flexible enough to cater to the needs of everyone.
- Inclusive design improves the product for everyone.
A Peek into the Future of A11y
In future iterations of A11y, the following will be included:
- Increasing the speed of text reading and exploring the Importation of text pictures under the text scanner mode.
- Making emergency calls to an A11y community and getting external aid from volunteers within that community.
- Identifying and finding people through facial recognition.
About Me
Hi, I’m David.
I’m an open-minded designer that creates with simplicity, usability, and modern aesthetics in mind. My creativity is limitless and it pushes me to explore new boundaries. I am passionate about accessibility design projects because working with diverse users energizes me and I love bringing their voices to the table!
To keep this article short, I only provided some of my findings. If you want to learn more about my study, send a mail or connect with me on Linkedin or Twitter.
