What Do We Mean by “Accessibility Research”?
In this post, we summarize the findings of our research paper “What Do We Mean by “Accessibility Research”? A Literature Survey of Accessibility Papers in CHI and ASSETS from 1994 to 2019”, which was accepted to the ACM Conference on Human Factors in Computing Systems (CHI) 2021. This paper was authored by Kelly Mack, Emma McDonnell, Dhruv Jain, Lucy Lu Wang, Jon E. Froehlich, and Leah Findlater.
Accessibility research is growing in popularity. Decades after the passing of key rights legislation for people with disabilities (e.g., ADA), this increased attention positions accessibility as an increasingly important and topical research area for the field of Human-Computer Interaction.
In a time of such considerable growth, it is important to reflect on the accessibility community’s research norms and trends. Our analysis of 26 years of accessibility research in CHI and ASSETS presents core trends from the accessibility community such as: what are the stated goals of the research (e.g., increasing digital accessibility), and what research methods are used, including study design decisions such as sample size and study location?
Our results revealed a heavy focus on people who are blind or have low vision (BLV). We found that people with disabilities or older adults appear in average samples of 13 participants per paper. While participants with disabilities or older adults were almost always engaged with in papers, people without disabilities (e.g, specialists, teachers, care givers) were sometimes included to serve as stakeholders, proxies, or as performance comparison points. We intend for this work to serve as a guide to support newcomers to the accessibility community and a reference and conversation starter for existing community members — for example, discussing what methods and sample sizes are appropriate for studies with different foci and goals, and the roles of ability-based comparisons and proxy use in accessibility research.
Methods
We conducted two studies. First, we qualitatively analyzed all ASSETS papers and all “accessibility-focused” CHI papers from 2010 to 2019. Accessibility-focused CHI paper candidates were identified by the use of the key terms “access-”, “assistive tech-”, “disab-”, and “impair-”, allowing for stemming (e.g., disab- matches disability and disabled), then verified manually by the authors. Our second study looks at a broader dataset of the same venues from 1994–2019, and we used quantitative methods to understand temporal trends within accessibility research. Through our qualitative coding process we developed 10 themes of interest, which, along with the temporal trends, are detailed in the paper. We summarize a subset of key findings from our qualitative study below.
Who does accessibility research study?
The HCI accessibility community overwhelmingly focuses on people who are blind or low vision (43.5% of papers), followed by people with motor or physical disabilities (14.2%) and people who are d/Deaf or hard of hearing (DHH) (11.3%). People with cognitive impairments, older adults, Autistic individuals, and people with intellectual or developmental disabilities were each found in less than 10% of papers. Few papers studied multiple disabilities in the same paper (7.1%) or people with multiple disabilities (<1.0%), which as Hofmann et al. point out, are common, key cases to include in accessibility research. Without considering the full range of people’s disabilities and other identities, accessibility research will fall short of including key, often multiply-marginalized, groups.
What methods does accessibility research use?
The accessibility community almost always employs user studies in research (94.3% of papers). The most popular study methods included interviews (42.1% of user study papers), usability testing (41.7%), and controlled experiments (34.6%). On the other end of the spectrum, methods that are generally well-suited for small samples sizes — often a concern with accessibility research — were used infrequently: case study (4.0%), focus groups (5.9%). Often, studies employed multiple methods (56.4%). User studies most frequently occurred in laboratories (32.1%) or in locations participants visited frequently (34.1%, e.g., their homes or workplaces). However, 39.6% of papers had at least one study where its location was unclear, which limits the ability of a reader to replicate and judge the validity of a study.
Who participates in studies and how are they engaged?
Almost all papers included disabled or older adult participants (90.1%), and the other 9.9% often engaged caregivers or other specialists (e.g., doctors, therapists, teachers). The median number of disabled or older adult participants recruited per paper was 13, which is only somewhat lower than the broader field of Human-Computer Interaction’s median sample size of 15–16 participants in lab, field, and interview studies. However, certain subcommunities had considerably smaller samples; for example, the median sample for autistic participants was 9. These numbers can offer a reference for researchers and reviewers to put samples sizes in context.
Papers often engaged nondisabled participants in the form of caregivers (9.4%), specialists (17.0%), or nondisabled people with no specific affiliation with disability (23.1%). We further examined how these nondisabled participants were engaged as proxies or as comparisons.
A proxy in accessibility research is when a person who is not disabled speaks about the thoughts or beliefs of a disabled person. Proxies were used in 8.0% of papers that performed user studies, often in conjunction with disabled or older adult participants (84.2% of proxy papers). Most papers did not specify why proxies were used. The few that did often cited communication difficulties as motivation for using proxies. Proxies were most often used in papers with participants with autism (22.6% of autism papers), cognitive impairments (17.4% of cognitive impairment papers), or intellectual or developmental disabilities (21.4% of IDD papers). On the other hand, no DHH-focused papers used proxies. These disparities in the use of proxies across user groups are cause for reflection.
Nondisabled participants were also sometimes used as a basis for comparison with disabled or older adult participants. Most frequently, comparisons were made to establish a “baseline” or “control group” for an experiment. As an example of establishing a baseline, one paper investigated whether touch screen interaction “reduce[d] the performance gap between older and younger adults” compared to using a mouse. Additional reasons for comparing the performance of disabled and older adult participant with nondisabled people was to find opportunities for collaboration (e.g., seeing if a cited person can help train a machine learning model for a blind person) or for understanding experiences with a technology across all ability levels (e.g., a college course captioning system will affect both DHH students and hearing students). Though comparisons are commonly used in accessibility research, our work highlights the need to examine the rationale for their use and to weigh the benefits and potential harms in any given case.
Key insights and areas of growth
While BLV research is receiving considerable attention, other populations are infrequently studied. People who were blind or low vision were studied far more frequently than people with motor impairments (43.5% vs 14.2% of papers, respectively), whereas people with motor impairments are the most represented disability in the world. The world is seeing increasing diagnoses of autism and intellectual or developmental disabilities, though these user groups only appeared in 6.1% and 2.8% of papers respectively. Similarly, few papers focused on people with multiple disabilities (e.g., someone who is autistic and blind) and groups of multiple people with different disabilities (e.g., a blind person and a deaf person). Each of these communities and scenarios serves as a potential growth area for accessibility research. Finally, we encourage researchers to consider adopting an accessibility lens to investigate problems faced by people with chronic illnesses or mental health related disabilities, as few papers encompass these groups.
A common criticism of accessibility work is the small sample size. Our work quantifies the difference in median participants engaged in papers between accessibility-focused CHI and ASSETS papers and broader HCI papers. While the overall median sample size of people with disabilities is close to that of general CHI papers (13 versus 15–16), some groups had smaller medians (e.g., autism: 9). Small sample sizes have some benefits. For example, repeated engagement with large samples of people from small communities or those sensitive to fatigue can place a substantial burden on participants. At the same time, we were surprised not to see more use of methods tailored to smaller sample sizes (e.g., case studies, single-subject experiments). To further reduce the burden of research on disabled or older adult participants, researchers can consider engaging with existing data (e.g., blogs, podcasts) by people with disabilities or older adults, with permission. Moreover, when reading accessibility papers, we encourage readers to consider if broader human-computer interaction norms (e.g., norms around sample size) are appropriate for a given research task and community.
Finally, we found trends that, without care, can reinforce ableist beliefs. For example, proxies were used frequently with people with autism, cognitive impairments, or intellectual or developmental disabilities. Moreover, researchers rarely discussed reasons for using proxies, and in the few cases where they did, often cited communication differences. We encourage researchers to carefully consider why proxies are required and develop studies that engage in communication that disabled or older adult participants are comfortable with before turning to proxies. If proxies should be used, consider triangulating the data collected through proxies with data collected from people with disabilities or older adults. Similarly, when papers use nondisabled participants as a “norm” or “baseline” to compare with disabled or older adult performance, it reinforces ableist beliefs that being nondisabled is the “goal” which all disabled people should try to achieve. For both use of proxies and ability-based comparison, we recommend that researchers discuss the rationale and tensions behind using these methods in their papers.
We hope that this work spurs conversation about the norms used in the ACM accessibility community. We invite you to read our paper for more details about accessibility research at CHI and ASSETS.
