Using AI to improve Non-verbal Communication

1 in 36 children in the U.S. have autism.

25-30% of this autist population are nonverbal or minimally verbal.

Current solutions for these communication challenges include flashcards and technology such communication boards or speech-generating devices; including the tablet shown below.

Image Source

Tablet-based devices, especially iOS devices (i.e., iPad) were found to be highly effective in increasing the communication skills of individuals with autism and developmental disabilities, based on the results of 15 studies.

Artificial Intelligence (AI) will provide massive improvements to these communication devices. AI will be able to take in information(user interactions, school curriculum, teacher or parent notes, Individualized Education Plans (IEP) information, etc.) and constantly adjust the experience to meet the specific needs of each person.

Generative UI — where the software interface is partially or fully generated by AI, not by designers — will be especially impactful.

Generative UI will allow the communication device to adapt in real-time, to best serve the person's needs.

This includes

1. Showing the most relevant words or phrases, based upon time of day, location or observed user behavior. This will improve the experience for things like finding the most commonly used words, communicating food selections during lunch time or describing symptoms during a doctors visit.
2. Dynamically adjusting the interface to make it easier to form sentences, further adjusting based upon a learner’s development. This can help students learn to gradually communicate with more complex sentences.
3. Automatically expanding vocabulary as language skills develop, including the selection of words of concepts that are most relevant to present to the learner.
4. Leveraging facial analysis to recognize emotions, and provide streamlined access to communicate these emotions. For example if frustration is the detected, the tool suggests calming phrases like “I need help” or “I am upset.” This will improve the ability of nonverbal people to communicate emotions during times of stress.
5. Automatically converting real-world instructions, this will allow parents, teachers and caregivers to convert speech into relevant and suitable symbols or phrases for non verbal communication.

By leveraging AI to adapt the interface in real-time, generative UI technology ensures that non-verbal individuals have access to the most relevant and effective communication tools, enhancing their ability to express themselves and interact with the world around them.