Genie: Automatically adding mouse, keyboard, speech, and other input to websites
Amanda Swearngin, one of my newer Ph.D. students, just presented her work on Genie at CHI 2017, in collaboration with me and her co-advisor James Fogarty. Genie is a clever technique that applies program analysis to reverse engineer a model of any interactive website’s commands, then uses that model to create alternative interfaces for accessing those commands via other input modalities such as keyboard, mouse, or speech. The really cool thing about this work is that web sites don’t have to be built for Genie: all of this works without modification, without the coordination of website developers, and continues to work as web sites evolve.
Here’s a demo:
What’s exciting about this work is that it could allow the entire interactive web to be accessible to people with diverse abilities without requiring web developers to design for diverse abilities. Even better, if developers do design for diverse abilities, Genie works even better, extracting even more meaningful command metadata.
Read our publication for more details.
