Seeing Beyond Sight

How insights into blindness can contribute to AI innovation.

Scrolling through news feeds, it seems that there are near constant reports of groundbreaking advances in the field of Artificial Intelligence (AI), but most people may find it hard to see the relevance to their daily lives.

Do these amazing breakthroughs offer us much above what we can already do for ourselves?

The picture above on the left shows a photo of a hot dog in a paper fast food carton on that is labeled “Hotdog!” On the right is a picture of a shoe in the same sort of paper fast food carton labeled “Not Hotdog!”

Consider, for example, “Not Hotdog” by SeeFood Technologies at the iTunes Store, or read more about how it works from a previous post by Nick Kasten titled “Taking Selfies (and More) to the Next Level with Open Source Deep Learning Models.” “Not Hotdog” was an image recognition application featured on the television show “Silicon Valley.” Parodied on the show to illustrate the prevalence of “solution-without-a-problem” tech pitched in the Valley, the “Not Hotdog” app makes one wonder: Who would find this useful?

Asking who our research can serve is important in aligning technical breakthrough with real-world use, helping us to make real, relevant advances that benefit society. This article discusses how people who experience blindness could help further the development of AI technologies. In the field of software accessibility for people with disabilities, “people are disabled to the extent that society creates accessibility barriers” [5]. For someone who experiences blindness or visual impairment, augmenting visual understanding can be a critical tool for independence.

A Second Pair of Eyes

Chris, a Rehabilitation Instructor at the Oregon Commission for the Blind, sits with her hands folded on her desk, where a keyboard, laptop, and abacus also rest. Photo credit: Feral Zen.art

“The iPhone has been a real game changer. We have apps to read barcodes, we have apps to read text. I can have the camera take a picture of a business card… I can sort through my junk mail. I could also have AIRA help me do that, my visualization service. Other than driving a car, I can’t think of anything that I can’t do. I can work on my website, I can decorate a cake. I made a Jello salad for the 4th of July and I decorated it with a flag made of blueberries,” shares Chris Cooke, a Rehabilitation Instructor at Oregon Commission for the Blind, during a visit to the Commission in January, 2020.

The AIRA app Chris mentions is not an automated AI application. AIRA is a subscription service that uses highly-trained people who can see the user’s surroundings through their mobile phone camera. While AI application vendors extol the high accuracy of their systems’ image recognition capabilities, the need for a human-powered service like AIRA shows that these automated systems are still inadequate to reliably serve people who experience blindness.

How could people who experience blindness improve the reliability of these systems? What can people who experience blindness teach the sighted about seeing?

Unsightly AI

The image recognition systems used in artificial intelligence are essentially complex probability models that indicate the likelihood that a given image matches a known category. For a system to know these categories, it must be trained on large amounts of labeled data. For the most part, the system can only make predictions based on what it has already seen. You can learn more about AI and images in another blog written by Nick Kasten last year on Artistic Style Transfer with deep learning.

Most applications intended to assist people who experience blindness are not trained on data created by nor relevant to people who experience blindness [5]. A photograph provided by a user who experiences blindness is very different than one by a user who does not. The information these users wish to know and their contexts are also very different. These variables are not typically taken into consideration when developing applications meant to assist users who experience blindness.

In practice, “blind” accessibility testing is commonly done by blindfolding sighted users [7]. This method demonstrates an assumption that the only difference between people who experience blindness and those who do not is that people who experience blindness can’t see. This assumption is not only wrong, but it is disrespectful and demeaning.

Sighted Assumption #1: People who experience blindness can’t see

Blindness isn’t a condition of complete lack of sight. People who experience blindness may have some partial vision or may be able to detect changes in light. Human-Computer Interaction research typically categorizes people who experience blindness as: Congenital Blind (ie, from birth), Late-blind (lost eyesight later in life), and Very Low Vision (having some limited eyesight) [3].

Humans are complicated and diverse; we all experience the world so differently. Some people are better at learning visually, others may prefer reading aloud, yet others may need hands-on experience. People who experience blindness are also like this; there isn’t just one kind of “blind” person.

“[People who experience blindness] vary greatly in their educational experience and in their perceptual skills. We have tests of spatial skills for sighted persons, since we are aware of their great variability. However, we don’t have these norms for [persons who experience visual impairments]” [3].

Char, a Rehabilitation Instructor at Oregon Commission for the Blind, stands in front of a bookshelf in her office. Photo credit: Feral Zen.art

Sighted Assumption #2: We only see with our eyes

When considering computer accessibility tools for people who experience blindness, most people only consider text to speech or screen reader tools like VoiceOver or JAWS. While useful, these tools are not universally ideal. Voice can be invasive, disorienting, and not particularly efficient. Braille, a haptic reading system developed by Louis Braille, on the other hand, is quiet, simple, and very efficient [8].

Julie, Assistive Technology Specialist at the Oregon Commission for the Blind’s technology center, demonstrates how to use Excel with a Braille display. Photo credit: Feral Zen.art

Refreshable Braille displays work with computers and are also built into standalone mobile devices that can sync with other devices via Bluetooth like the iPhone. According to Julie, Assistive Technology Specialist at the Oregon Commission for the Blind, these types of devices are “why 80% of people who have Braille skills have jobs. Braille isn’t just the big books in the library that take up lots of room; it’s devices like this that can connect to a computer.”

Chris, a Rehabilitation instructor at the Oregon Commission for the Blind, reads a story from Aesop’s Fables on a Braille mobile interface. Photo credit: Feral Zen.art

Notice in the image above how Chris uses both hands to read. This is a key reason why blindfolding sighted people does not equate to someone who experiences blindness. “Sighted subjects are much more likely to use a single finger of one hand to explore patterns” [9].

Debra, a Rehabilitation Instructor at the Oregon Commission for the Blind, uses a talking label-reader device to determine what’s in a particular bottle. The pen reader can be programmed to recognize certain labels and the user can record their own voice (or someone else’s) to say what it is. The instructor described how her grandchildren use it to leave her surprise messages. Photo credit: Feral Zen.art

In a study on developing tools for conveying information and graphs using natural language, people who experience blindness were observed as using twice as many commands, checking the starting point more often, and using different navigation strategies than sighted users who were blindfolded [1]. Additional research has shown that “[people who experience blindness] tend to select a very efficient strategy for feeling large displays, and this takes the form of using two hands for exploration. This speeds up the acquisition of information and places less of a burden on memory” [3].

Many studies to assess visualization in people who experience congenital blindness use tactile-friendly (often raised-line or textured) 2-dimensional images. Some of these studies have concluded that poor performance in these assessments suggests that people who experience blindness have impaired abilities in the areas of visual-spatial cognition.

In daily life, when we grab hold of something or hug someone, our hands are touching the back. When we use our eyes, however, we know an object from the front. A research participant told one researcher that they were aware that sighted people “see half of a tree”; they described themselves as “imagining the whole tree” [3]. Studies considering these factors suggest tactile image recognition actually works best with the backs of objects [6].

Besides only representing one way of seeing, the two-dimensional rendering of images has another challenge. While software developers accustomed to images and computer screens may take this rendering for granted, it can be counter-intuitive to many people, including some who experience blindness. Research suggests that this isn’t due to any lack of cognitive capability, rather it is more likely merely a lack of education [3]. Perspective, or realistically rendering three-dimensional subject matter in two-dimensional space, is a recent human invention from the Renaissance. People aren’t born with the ability to see these kinds of pictures, it’s something we have to learn how to see.

A cube in two-point perspective, Image via Wikipedia.

Two different projects of a stack of two cubes, illustrating oblique parallel projection foreshortening (A) and perspective foreshortening (B) Image via Wikipedia.

Optical illusions also demonstrate the problem with assuming that we only see with our eyes. Which line is longer? [4]

Esref Armigan, a Turkish painter who experiences blindness, with a painting he made as part of a Volvo marketing campaign. Image via Wired.

Blind and sighted people use many of the same devices in sketching their surroundings, suggesting that vision and touch are closely linked. The text in the image reads, “Blind artists, such as Tracy (above), rely on their sense of touch to render familiar objects. Tracy lost all sight to retinal cancer at the age of two, but by feeling the glass, she determines its shape. By rubbing the paper, placed on a piece of felt, she knows where her pen has scored the page and left a mark.” via Art Beyond Sight.

Insights for Inclusion

In accessibility and inclusion, whether the topic is software interfaces, datasets, or other contexts, typically “disabled” users are lumped together under one umbrella based on deviation from an ideal human standard. From this lens of universal access, accessibility is an impossible dream (or possible nightmare). What if instead of categorizing based on imperfections, we started from a place of seeing people as we are?

Several researchers working in the area of Accessibility and AI, including IBM and Microsoft, have proposed recommendations for inclusion [2][10]. Their recommendations are highlighted below along with more concrete suggestions pertaining to people who experience blindness. These suggestions could also be adapted to other accessibility use cases.

Recommendation #1: Identify ways in which inclusion issues for people with disabilities may impact AI systems

Here’s how:

• Focus on how someone sees — Instead of constructing “blind” or “non-sighted” personas that emphasize lack of eyesight, consider archetypes based on preferred modes of sensory I/O (Input/Output). The reason why a person doesn’t prefer a visual mode matters less when we focus on what they do prefer. Use data gathered with people using non-visual modes of I/O, including people who experience blindness. Consult with local resources for people who experience blindness, including state or city services, Braille reading centers, or even local public resources such as the public library. Remember, there is no “typical” blind user, so even a small sample size can provide really rich insights.
• Let people who experience blindness help you — Here are two ways to initiate such a relationship:

1. Learn how to use accessibility features and devices — Have you ever used any special accessibility features for people who experience blindness? The option most people think of is screen-reader software such as VoiceOver (Mac) and JAWS (Windows). Many people with blindness prefer quieter haptic I/O. Braille displays can dynamically change text as a person reads as indicated by their hand position. These displays can also provide feedback when manipulating text, such as a “blinking” cursor.
2. Make your work facilities more accessible for people who experience blindness — Empathy is cultivated through understanding, and the best way to cultivate understanding is through first-hand experience. Vocational training facilities like the Oregon Commission for the Blind offer consulting services for companies that would like to be more accessible to people who experience blindness. They can also recommend resources for sighted people looking to gain these special abilities.

Recommendation #2: Understand the limits of bias mitigation and develop new methods to address their shortcomings.

Here’s how:

• Use interdisciplinary teams and open-source bias mitigation tools — Interdisciplinary teams are vital for providing different perspectives on complex problems. Many people working in the field of Accessibility and User Experience, for example, have backgrounds in social science. Open-source bias mitigation tools not only ensure transparency, but allow you to submit feedback and improvements when they come up short. IBM’s AI Fairness 360 project offers a comprehensive set of metrics and algorithms to assist with bias mitigation. They also welcome contributions to improve the project’s usefulness.
• Actively curate inclusive data sets — Create benchmark datasets to support replication and inclusion. Don’t forget to consider the complex ethical issues that creating such datasets for vulnerable groups might involve (this is where an interdisciplinary team comes in handy). The next article in this series will feature Viz Wiz, a research project that curates datasets originating from people who experience blindness in order to improve computer vision technology.

Innovation doesn’t happen from following the pack. We innovate when we go back to square one and turn our assumptions on their heads. The more we limit ourselves to only seeing each other in terms of what we lack, the more we’re not really seeing ourselves at all.

Woodshop at the Oregon Commission for the Blind. This is all standard equipment. Photo credit: Feral Zen.art

Acknowledgement

I’d like to offer many heartfelt Thank-you’s to the Oregon Commission for the Blind for sharing their facilities and their distinct abilities.

The front of the Oregon Commission for the Blind office. Photo credit: Feral Zen.art

Resources

• Oregon Commission for the Blind
• Data Asset eXchange (DAX)
• IBM AI Fairness 360

References

[1] Ferres, Leo, Gitte Lindgaard, Livia Sumegi, and Bruce Tsuji. “Evaluating a tool for improving accessibility to charts and graphs.” ACM Transactions on Computer-Human Interaction (TOCHI) 20, no. 5 (2013): 28. Retrieved from https://dl.acm.org/doi/abs/10.1145/2533682.2533683 (requires ACM subscription)

[2] Guo, Anhong, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, and Meredith Ringel Morris. “Toward Fairness in AI for People with Disabilities: A Research Roadmap.” arXiv preprint arXiv:1907.02227 (2019). Retrieved from https://arxiv.org/abs/1907.02227

[3] Heller, Morton A., Melissa McCarthy, and Ashley Clark. “Pattern perception and pictures for the blind.” Psicológica 26, no. 1 (2005): 161–171. Retrieved from https://www.researchgate.net/profile/Morton_Heller/publication/26421660_Pattern_Perception_and_Pictures_for_the_Blind/links/0912f51198d138f048000000/Pattern-Perception-and-Pictures-for-the-Blind.pdf

[4] Heller, Morton A., and Tamala D. Joyner. “Mechanisms in the haptic horizontal-vertical illusion: Evidence from sighted and blind subjects.” Perception & Psychophysics 53, no. 4 (1993): 422–428. Retrieved from https://link.springer.com/content/pdf/10.3758%2FBF03206785.pdf

[5] Morris, Meredith Ringel. “AI and Accessibility: A Discussion of Ethical Considerations.” arXiv preprint arXiv:1908.08939 (2019). Retrieved from https://arxiv.org/vc/arxiv/papers/1908/1908.08939v1.pdf

[6] Newell, Fiona N., Marc O. Ernst, Bosco S. Tjan, and Heinrich H. Bülthoff. “Viewpoint dependence in visual and haptic object recognition.” Psychological science 12, no. 1 (2001): 37–42. Retrieved from https://journals.sagepub.com/doi/abs/10.1111/1467-9280.00307 (requires institutional access)

[7] Sears, Andrew, and Vicki Hanson. “Representing users in accessibility research.” In Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 2235–2238. ACM, 2011. Retrieved from https://dl.acm.org/doi/10.1145/2141943.2141945 (requires ACM subscription)

[8] Shinohara, Kristen, and Josh Tenenberg. “Observing Sara: a case study of a blind person’s interactions with technology.” Proceedings of the 9th international ACM SIGACCESS conference on Computers and accessibility. ACM, 2007. Retrieved from https://www.researchgate.net/publication/221652126_Observing_Sara_a_case_study_of_a_blind_person's_interactions_with_technology

[9] Symmons, Mark, and Barry Richardson. “Raised line drawings are spontaneously explored with a single finger.” Perception 29, no. 5 (2000): 621–626. Retrieved from https://www.researchgate.net/profile/Mark_Symmons/publication/12330124_Raised_line_drawings_are_spontaneously_explored_with_a_single_finger/links/0c9605214626537b1d000000.pdf

[10] Trewin, Shari, Sara Basson, Michael Muller, Stacy Branham, Jutta Treviranus, Daniel Gruen, Daniel Hebert, Natalia Lyckowski, and Erich Manser. “Considerations for AI fairness for people with disabilities.” AI Matters 5, no. 3 (2019): 40–63. Retrieved from http://sigai.acm.org/static/aimatters/5-3/AIMatters-5-3-09-Trewin-accesible.pdf