Voice assistant + screen reader = the best of both worlds

Phil Siarri
Aug 20 · 3 min read
Image of geometric shapes
Image of geometric shapes
Image by Alexei Hulsov from Pixabay

Researchers at the University of Waterloo (Canada) have developed a new voice assistant that enables individuals with visual impairments to get web content quickly and (relatively) effortlessly from smart speakers and similar devices.

Merging voice assistants with screen readers

Through a collaboration with tech giant Microsoft, the researchers developed a framework that merges elements of voice assistants with screen readers to create a tool that makes web searches easier. The tool was named Voice Exploration, Retrieval, and Search (aka VERSE).

“People with visual impairments often rely on screen readers, and increasingly voice-based virtual assistants, when interacting with computer systems,” stated Alexandra Vtyurina, a PhD candidate in Waterloo’s David R. Cheriton School of Computer Science, who took part in the study during her internship at Microsoft Research. “Virtual assistants are convenient and accessible but lack the ability to deeply engage with content, such as read beyond the first few sentences of an article, list alternative search results and suggestions. In contrast, screen readers allow for deep engagement with accessible content, and provide fine-grained navigation and control, but at the cost of reduced walk-up-and-use convenience. Our prototype, VERSE, adds screen reader-like capabilities to virtual assistants, and allows other devices, such as smartwatches to serve as input accelerators to smart speakers” she added.

“People with visual impairments often rely on screen readers, and increasingly voice-based virtual assistants, when interacting with computer systems,” Alexandra (Sasha) Vtyurina, PhD student at University of Waterloo

A new form of input

The main input method for VERSE is voice; hence, users can say “next”, “previous”, “go back” or “go forward”. VERSE can also be paired with an app, which runs on a smartphone or smartwatch. These devices can act as “input accelerators”, similar to keyboard shortcuts (rotating the crown on a smartwatch advances VERSE to the next search result, section, or paragraph, depending on the navigation mode).

“VERSE: Voice. Exploration. Retrieval. Search.” by Microsoft Research, YouTube.

53 visually impaired web searchers were surveyed throughout the study. More than half reported using voice assistants several times a day, and a vast array of devices (smart speakers, phones, as well as smart TVs). The data gathered from the survey was utilized to design of a prototype of VERSE.

“At the outset, VERSE resembles other virtual assistants, as the tool allows people to ask a question and have it answered verbally with a word, phrase or passage,” explained Vtyurina. “VERSE is differentiated by what happens next. If people need more information, they can use VERSE to access other search verticals, for example, news, facts, and related searches, and can visit any article that appears as a search result. For articles, VERSE showcases its screen reader superpowers by allowing people to navigate along words, sentences, paragraphs, or sections.”

This is a great “tech for good” use case in my opinion, and I’m definitely looking forward to new developments. 🙏🏽


Discovering the optimal lifestyle of the future.

Phil Siarri

Written by

Founder of Nuadox | Tech & Innovation Commentator | Industry Analyst | One of #TRFinRiskCanada40 | More about me> psiarri.xyz


Discovering the optimal lifestyle of the future.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade