Blind Accessible Keyboards — Design Diaries
Objective Comparisons between Gesture and Exploration Based Touchscreen Keyboards for the Blind
This paper was first published in the Asia Pacific CHI of 2013. Here is a link.
Abstract
We report the results of a short term experimental study on blind participants to evaluate their performances on two different typing techniques. BrailleTouch, a gesture based text typing technology, is objectively compared to the TalkBack-enabled Android stock keyboard which requires touch exploration to type. We found that on an average BrailleTouch was almost twice as fast as the Android stock keyboard for typing each character. Also these experiments clearly establish that accuracy of the BrailleTouch technique significantly dwindles as the number of dots in a Braille cell increase due to lesser motor control over the ring finger (or the digitus medicinalis).
Introduction
The visually impaired are currently unable to use smartphones mainly due to the lack of tactile cues on touchscreens to navigate and type. Hence, Touch screens remain largely inaccessible to blind users since they must adopt error-prone compensatory strategies to use them or find accessible alternatives [5]. Also speech to text is not very efficient in handling different languages, colloquial terms and proper nouns for the blind to interact while attempting to input text. Smartphones, if equipped with better accessibility for typing text, can prove to be valuable tools for the blind to navigate streets, send text messages or emails, search and possibly usher in some revolution in social networking for the visually impaired.
According to a report by WHO in 2010, there were about 39 million blind people in the world. More than 90% of them lived in developing countries and India had one of the largest population of blind people of about 8 million [4]. But only 5% of them receive any kind of education or assistive technologies needed for education. If current commodity smartphones had intuitive software accessibility then the blind could be empowered with superior computing capabilities with the least cost. This was one of the key motivations behind attempting to understand the current state of the art in smartphone accessibility for the blind from an Indian blind user’s perspective.
BrailleTouch Keyboard
BrailleTouch is an eyes-free text entry technology for multi-touch smartphones. The key aspect of the technology is that it has fewer buttons than fingers. Thus, the user does not have to move the fingers around to find the correct sequences and combinations to type. Once placed, the fingers remain in the same position. This is crucial for eyes-free text input on a smooth surface, like a touch screen or a touch pad [2].
The six buttons on BrailleTouch spatially correspond to the mental map of the six cells in a Braille character as well as to the placement of the six fingers as shown in Figure 1. As the user types, BrailleTouch provides audio feedback for each selected character [3]. Users hold the BrailleTouch enabled smartphone with the screen facing away from them with two hands. They arrange their fingers in a one-to-one correspondence to a standard Braillewriter [3].
Talkback-enabled Android keyboard
The TalkBack keyboard is an on-screen keyboard that supports touch exploration along with synchronized spoken and auditory feedback. This means visually impaired users can enter text when using Android devices that don’t sport a physical keyboard.
In this mode of text entry, users can hear keys as they explore the keyboard along with an auditory icon; picking up their finger types the last key they explored. Typing produces a distinctive key-click and every space key press speaks out the latest word that has been typed. The on-screen keyboard occupies about 1/3rd of the bottom area of the smartphone screen as shown in Figure 2.
Experimental Methodology
Our goal was to compare two very different methods of text typing for the blind and provide recommendations based on our comparisons. Our research questions aiming at this goal were:
RQ1: How do blind individuals perform objectively when presented with the gesture based and exploration based input mechanisms?
RQ2: What real problems do the blind people in India face while typing without any tactile feedback with Braille cells as input characters versus the QWERTY keyboard characters?
Participant Demographics
In the experimental study at Blind Peoples Association, Ahmedabad (BPA) we recruited five completely blind participants of which one had to drop out after the learning phase. All the participants were males and used Laptops or PCs in their daily lives and jobs using a popular screen reading software (JAWS). Thus they were well versed in the QWERTY keyboard structure and also well adapted to audio feedback whilst typing.
The average age of the participants was 34.5 years (S.D. of 10.16 years). Three participants of the four completed their schooling from Kindergarten to high school (K-12) in Braille. Only one participant used a smartphone with a virtual keypad and was well versed with the TalkBack accessibility features of Android smartphones. Other participants used the Nokia E5 phone with the TALKS accessibility feature. All of them regularly used email and web browsing to access information using their phones and laptops.
Additionally each participant had at least 10 years of education in Braille although most of them rarely used Braille for typing in their daily lives anymore. Although P1 (Table 1) did use a refreshable Braille display computer occasionally apart from his accessible desktop computer. Participants P1 and P2 were also blind computer teachers and taught blind adult students some professional software according to student needs.
Experimental Procedure
The Experiment was performed by creating an experimental application which essentially logged data into a phone memory storage which can be easily retrieved later. The Samsung Galaxy Nexus smartphone (Android version 4.2.2 Jelly Bean) was used in each experiment to compare both techniques of text input. Portrait orientation mode was used for Android QWERTY keyboard while for the BrailleTouch keyboard Landscape orientation mode was used in all experiments. These orientations were chosen since the natural way for typing on Qwerty keyboards is the portrait mode and BrailleTouch was designed specifically for the landscape mode. Also these orientations were fixed since the auto-rotation of the screen would have confused our blind participant.
Figure 3 shows the entire scheme of the experiments conducted with our participants. It consists of three sections namely Experimentation, Data Collection and Analysis.
First we introduced the participants to a new system of text input and the various gestures through which they would be able to input text. They were also taught the various feedback cues through which they could understand if the letter or word they had entered was what they intended to type.
Next, in the learning phase of the experiment we let them explore the system thoroughly without time constraints. We also kept logging the data in order to analyze the learning curve of our participant. Since, the participants were allowed to explore both systems without any target sentence the learning data was not used in the final analysis of the keyboards.
Following this the participants were given a typing task and were told to focus only on accuracy and no time constraints were given to them to complete these typing tasks in order to ensure completion of the typing tasks. The task would entail typing an English sentence (without numbers, uppercases or punctuations) like “the quick brown fox jumps over the lazy dog”. Assistance in spellings was provided whenever they requested it since English was their second language and different participants may have had different levels of fluency in English. In order to remove any variation or bias introduced by differing levels of English capability producing different typing speeds, we opted to provide them with the spellings, so that the results would solely depend on how well they were able to type with the two methods being tested.
This entire experiment was repeated by introducing the participant to a different system of text input. Finally, a subjective survey was taken from each participant at the end of each experiment.
Text data collection was done in the background by the experiment application and stored in separate files for analysis. Pictures of the participants were taken to obtain insights on the interaction of the blind with the touchscreen device (as shown in the data collection section in Fig. 3).
The data collected was analyzed to obtain the speed and accuracy of the participants in each technology of input. We also gained insights on the learning curves and the common errors made in both methods. This analysis has been detailed in the following section.
Data Analysis
Keystroke time is the time require for the participant to type a single character on each technique of text input.
These keystroke timings were logged in a key-log database which captured the timestamp both in milliseconds, character (or backspace) typed and the current string of characters in the text box.
A total of approximately 280 minutes of text typing data was collected from the Typing tasks from the participants apart from learning phase data. The participants could type an average of 3.98 words per minute (wpm) using BrailleTouch while they could only type at an average of 2.17 wpm on the Android keyboard.
Error Inclusive Keystroke Time Calculation
The raw data collected was then processed to obtain the time required for each keystroke, the intended target character and the error involved (keystroke time, Target character and Match respectively as illustrated in Fig. 4).
The keystroke time gives us only the time required to type exactly that character but does not include the error correction time. Hence, we add those timings into the target character and obtain the ‘error-inclusive keystroke timing’. Hence, in the illustration shown in Figure 4 the keystroke time in order to type the character “h” is 3530 milliseconds in this case. While, the error inclusive keystroke timing to type “h” is 9910 milliseconds which is the summation of all the keystroke times that the participant incurred in order to correctly type the character intended, in this case “h”.
This Error Inclusive timing gives a clearer picture of the accuracy and the actual time required to type when both input techniques are objectively compared.
Observations and Results
Aggregated Keystroke Time Comparisons
When average keystroke timings were plotted for each character typed using BrailleTouch and compared with the keystroke timings for the TalkBack enabled Android keyboard we obtained a chart illustrated in Figure 5. This chart clearly suggests that the gestures in BrailleTouch are quicker than the exploration based Android keyboards. This can also be confirmed from the typing speeds obtained in table 3 for both techniques.
Thus the chart in Figure 5 clearly shows that BrailleTouch is faster in comparison to the Android keyboard in terms of performance objectively measured by time required to type each character.
Error Comparisons
Next we obtained a comparative chart (Figure 6) of average number of errors for both the text typing methods. In this chart we made certain interesting observations on the reason behind errors made in BrailleTouch. We observe that while typing BrailleTouch the participants made almost equal number of mistakes as the Android keyboard for Braille characters which do not require the 3rd or the 6th dot. As we incremented the number of dots the number of errors per character also increased. Hence the accuracy of BrailleTouch was extremely low for many characters which involved four or more dots to type as shown in Figure 10. Characters “v”, “w”, “y” and “z” have as much as approximately 1.5 errors per character typed while a vowel like “u” has more than 3 errors per keystroke!
Figure 6: Plot of the average number of errors per keystroke for each character in both techniques
This error in typing can be attributed to lesser motor control of participant in the ring finger (or the digitus medicinalis) which corresponds to the third and the sixth dot in the Braille characters for typing using BrailleTouch. Thus, as the dots in the Braille cell increase and consequently involved more usage of the ring finger the accuracy of BrailleTouch technique decreased significantly.
Aggregated Error-Inclusive Keystroke Time Comparisons
We then plotted the error inclusive average keystroke time for each character in both techniques and compared the results. Figure 7 shows the chart we obtain when the characters are ordered with increasing number of Braille cell dots involved. It is fairly similar to chart (Figure 5) obtained by just plotting the average keystroke times initially when the number of dots are low. As the number of dots increase we obtain a visible deviation of the error inclusive timings chart in Figure 7 from Figure 5. This observation is true even after removing the data of participant P4 who was more experienced in the Android smartphone keyboard.
Thus we can arrive to a conclusion that the Android QWERTY keyboard enabled with TalkBack is far more accurate and time saving than BrailleTouch if the number of dots involved in typing Braille is high. Also, since these experiments were performed over a shorter duration (4 to 5 hours for every Participant) we expect that these errors may be reduced by more experience with the BrailleTouch technology and hence would reduce the error-inclusive keystroke time.
Subjective feedback
The experiments also involved a subjective survey to assess these technologies of text typing as well; they gave us some really vital feedback on designing typing interfaces for Indian blind users.
All the Participants preferred the British male accent over the American Female accent of the Google Text-to-Speech output voice. Participant P4 reasoned that the British accent could clearly pronounce the native colloquial words (in Hindi and Gujarati) that they use in text messages and chats. Participants also found the backspace key hard to locate in the Android keyboard and suggested improvements would be required in that.
The curled fingers posture required for BrailleTouch phone was a bit difficult to maintain for the participants. They would have preferred the tablet version of adaptive BrailleTouch keyboard, as suggested by P1. The participants strongly suggested that both of these keyboards would require a better word editing tool to go back and correct misspelled words.
All Participants felt that they could perform faster had they been given a sufficiently long time (say a month) to regularly use the BrailleTouch keyboard.
Conclusions
Analyzing the data objectively in this study has led us to certain key observations in both text typing techniques.
Exploration based techniques (like Android QWERTY) are slower than gesture based techniques (like BrailleTouch). Although BrailleTouch does require a much higher amount of time to be adapted to in comparison to the standard QWERTY Android keyboard.
As the number of Braille dots increase for a particular character (and hence the number of fingers required to accomplish the gesture increase) the accuracy of BrailleTouch decreases. This lead to a fairly high error inclusive keystroke time for many characters in BrailleTouch. Lesser motor control over the ring fingers may be the cause of the high number of errors in letters t, u, v, w, y and z affecting the performance of participants on BrailleTouch.
This study clearly reveals that there are some key advantages and drawbacks in both methods of text typing. A longer term study and evaluation of these accessible keyboards would be required to assess if a hybrid approach between these two methods of text input may improve touchscreen typing efficiency of the blind. The experimental study described in this paper lays a strong foundation for such a longer term study.
Acknowledgements
We wish to thank all the participants in our experimental study and the Blind Peoples Association, Ahmedabad (BPA) for actively helping us in this project. The Computer and Digital Audio laboratory at the BPA have been instrumental in the success of this research project.
References
- Brian Frey, Kate Rosier, Caleb Southern, and Mario Romero. 2012. From texting app to braille literacy. In CHI ’12 Extended Abstracts on Human Factors in Computing Systems (CHI EA ‘12). ACM, New York, NY, USA, 2495–2500. DOI=10.1145/2212776.2223825 http://doi.acm.org/10.1145/2212776.2223825
- Frey, B., Southern, C., Romero, M., “BrailleTouch: Mobile Texting for the Visually Impaired.” Proceedings of Human-Computer Interaction International, HCII. Orlando: July 2011
- Mario Romero, Brian Frey, Caleb Southern, and Gregory D. Abowd. 2011. BrailleTouch: designing a mobile eyes-free soft keyboard. In Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services (MobileHCI ‘11). ACM, New York, NY, USA, 707–709. DOI=10.1145/2037373.2037491 http://doi.acm.org/10.1145/2037373.2037491
- Pascolini D, Mariotti SPM. Global estimates of visual impairment: 2010. British Journal Ophthalmology Online First published December 1, 2011 as 10.1136/bjophthalmol-2011–300539
- Shaun K. Kane, Jeffrey P. Bigham, and Jacob O. Wobbrock. 2008. Slide rule: making mobile touch screens accessible to blind people using multi-touch interaction techniques. InProceedings of the 10th international ACM SIGACCESS conference on Computers and accessibility (Assets ‘08). ACM, New York, NY, USA, 73–80. DOI=10.1145/1414471.1414487 http://doi.acm.org/10.1145/1414471.1414487
Some pictures from the study.
Do check out my other design projects on — aditsgupta.com
Say Hi! via Twitter or Email. I’d love to talk more about my work!
