Danger! Testing Accessibility with real people
As four people who are blind and care deeply about making the web more accessible, we strongly believe user testing should include people with disabilities. But when the results are misinterpreted, it can be dangerous. It can foster action that appears to benefit people with disabilities but ultimately do as much harm as good. This article that suggests that ARIA tabs are dangerous is a perfect example. While the methods and logic supporting it have perfect intentions and goals and were professionally executed, we believe its broadly sweeping conclusions are not only invalid but detrimental. Rather than accelerating progress on a path toward a more accessible web, we believe following the advice given in that article would take everyone on a needless and confusing detour.
We will explain problems we see with those conclusions and, perhaps more importantly, shed light on how to avoid similar pitfalls when analysing results from user testing.
Who are we?
We are Birkir Gunnarsson from Deque Systems, Bryan Garaventa from SSB BART Group, Lèonie Watson from The Paciello Group (TPG), and Matt King from Facebook. We are four blind people who access our computers, tablets and phones using a screen reader. Each of us works in the accessibility profession, and we are all involved in developing the ARIA specification at the W3C. We mention these things to be completely open about our level of technical knowledge and our obvious interest in promoting ARIA as a technology for making the world more accessible.
What are the article’s problematic conclusions?
The article that decries ARIA tabs as dangerous derives 2 recommendations based on user research:
1. ARIA mark-up on tabs creates barriers for some screen reader users so it is better to leave it off.
2. Some keyboard users expect the tab key to move focus to all interactive elements so it is better to put all tabs in the keyboard tab sequence.
The implementation proposed in the article still looks like a set of tabs from a typical Graphical User Interface (GUI) desktop application. And, for mouse users, it behaves like desktop tabs. But for keyboard and screen reader users, it feels more like an ordinary list of links with the exception that following a link doesn’t just take you some place, it magically causes a block of content to disappear and some new content to take its place.
How can we fault the simplicity of treating tabs as links?
That’s a good question. As links in the page’s tab sequence, anyone can operate them, and that is our common goal, right?
But wait, why do some people have the opportunity to perceive and operate those elements as tabs? Why don’t sighted mouse users also get something that looks and works like a simple list of links? Really, what’s wrong with a good old fassion list of links that serves as a table of contents for the page? Is it because there are benefits to a tabbed interface?
Indeed, web designers employ tabs because they are useful. They are fundamentally different from a list of links. And, much of their utility is derived directly from those differences.
We have concerns about the recommendations made in the article because:
1. Our data and experiences across many products tell us tabbed interfaces provide the same utility for typical screen reader and keyboard users as they do for sighted mouse users. In fact, in some ways, they are more valuable for people with disabilities.
2. Presenting tabs as links to screen reader users denies screen reader users some of the benefits of tabs.
ARIA tabs do in fact work very well with all popular screen readers.
3. Persisting the confusing pre-ARIA, web 1.0 practice of visually styling web content as a GUI without implementing long-standing and well-researched GUI keyboard conventions steepens the mountain people with disabilities must climb to get equivalent benefit from the web.
So, how do we reconcile these claims with the research findings that suggest otherwise? We will dig in to those findings and see. But first, we should explain why ARIA is essential to building equal access.
Why is ARIA so important?
The web has evolved into a software platform where most sites incorporate graphical user interface (GUI) elements, such as popup menus, tabs, trees, comboboxes, and dialogs. ARIA is a set of attributes that provide screen readers and other assistive technologies with semantic information they can use to render GUI elements in a way that is understandable.
Technically, it is possible to make even GUI elements screen reader and keyboard accessible with only the behaviors of basic HTML. That is sort of what WCAG 1.0 and the 2001 version of standards required by US section 508 forced on the world. For example, using the web 1.0 flavour of web accessibility, you could make a tree accessible by having users tab through every element, including the icons that expand and collapse branches. Those icons were just graphical links which change their name based on the state of the branch.
While such web 1.0 accessibility is often simpler to implement, and while it may appear relatively easy for a certain segment of users to learn, it does not come close to enabling universal, equal access in a world of GUIs. It is slow and cumbersome even for the most proficient assistive technology users. Tasks that sighted mouse users easily complete in seconds often take experienced screen reader users many minutes as they try to discover how the page is changing in response to their actions.
However, the most critical difference between web 1.0 accessibility techniques and ARIA is that web 1.0 techniques do not provide assistive technologies with the semantic information that distinguishes a link from a tree node or menu item or any other GUI element. Without this rich semantic information, assistive technologies are hamstrung. They cannot guide interactions. They cannot produce alternative renderings in service of their specific users’ limitations. This puts the most novice and most challenged users at the greatest disadvantage.
ARIA enables us to build universal web accessibility at scale because it attacks the problems of web usability for people with disabilities in a way that parallels how web engineers are solving them for people who do not have a disability, harvesting user expectations, paradigms, technologies, and lessons learned from the desktop GUI world. With ARIA, elements of an interface that look and act like a GUI for mouse users can have accessibility that not only builds on the well-proven aspects of desktop GUI accessibility but also provides the foundation for many needed enhancements. In fact, there are many ways in which web technologies, when supplemented with ARIA, provide a much better platform for accessibility than any desktop platform has ever provided.
Web GUIs are already pervasive and presenting barriers for people with disabilities. We urgently need to accelerate a web accessibility paradigm shift capable of unlocking the full value of those GUIs. ARIA is designed to support such a shift. We would go so far as to say that without a wildly successful ARIA standard, anyone who relies on assistive technology will struggle to have anything better than third class access to the internet.
Assistive technology developers have been giving people with disabilities a high degree of access to GUIs in desktop software for many years, but GUIs on the web were out of their reach until web developers began increasing adoption of ARIA, which became a standard in March of 2014. Admittedly, as a young standard, there are still some quirks. There are also a variety of challenges with transitioning the world to web 2.0 accessibility from web 1.0 accessibility. And, lurking behind all of that, there are still shortcomings in the way assistive technologies provide access to visually-centric GUIs, whether on the web or the desktop.
The article purporting that ARIA tabs are dangerous states that the quirks with ARIA support in screen readers combined with the expectations set by broad adoption of web 1.0 accessibility justify throwing out the ARIA approach to tabs. It says ARIA tabs, and perhaps other ARIA widgets, are not yet well enough supported in the real world, and that the only people who could possibly understand and benefit from such widgets are the kind of people who would write an article like this one!
Let’s examine those assertions, starting with the screen reader experience and the research findings offered in support of the thesis that ARIA tabs are dangerous.
How do screen reader users experience ARIA tabs?
We can best show how ARIA makes tabs understandable to screen reader users by demonstrating it in practice. The tabpanels in the following two videos look the same, but they are presented differently by the screen reader.
The tabs without ARIA example uses plain html, and tabs with ARIA example uses the same html with the appropriate ARIA roles and attributes added. Listen carefully to what the screen reader says.
1. Tabs without ARIA video
2. Tabs with ARIA video
In the first video the screen reader announces that there is a list containing a number of items, and that the list contains links that point elsewhere on the page. The screen reader also announces some content immediately below the list.
In the second video the screen reader announces that there is a collection of tabs, how many tabs are in the set, the name of each tab, and which tab is currently selected. It also tells the user where the content of the tabpanel starts, and provides a shortcut for moving to the tabpanel content.
What makes the ARIA version better for a screen reader user?
Without ARIA, screen reader users:
1. Do not know which tab is currently selected. You could adopt a web 1.0 technique and add off-screen text to the link name to indicate the selected link but that is not information that is understood by the screen reader, depriving the user of screen reader features that control how and when states are rendered. This is especially important to braille users.
2. Are not aware of the boundaries of the tabpanel. This too could be fixed with off-screen text, but again, that prevents screen readers from being able to provide features like skip to beginning and skip to end of container.
3. Lack awareness of new content or lose the tablist context. If activating a link does not move focus to the tabpanel content, the screen reader user will not be aware of the change. If it does move focus, the screen reader user has lost context, making it unclear how to return to the tab list. There are web 1.0 work-arounds for these issues as well, each with its own set of trade-offs.
4. Cannot easily discern that activating one of the links will cause one chunk of content to disappear and another to appear in its place. There are several scenarios where this is really important for the user to understand. For instance, the user could expect that the screen reader page search function will locate content from a previously viewed tab and then be confused when that search fails.
Those are some advantages of the ARIA tab semantics. Of equal or greater importance to screen reader users are keyboard navigation efficiencies. But, before we go there, we should talk about the screen reader user tripping points mentioned in that other article.
What about observations of screen reader users struggling with ARIA tabs?
Here is the first screen reader observation from the dangerous tabs article:
“Some of the non-visual users in our studies used the links list functionality in their screen reader software. When they did, the links with role=”tab” within the role=”tablist” element didn’t show up in the list. These elements have moved to a list of ARIA elements — a relatively new piece of functionality in screen readers, and one that some folks weren’t familiar with. So, these users weren’t able to find any of the content contained in the inactive tabs.”
There is something missing here. The conclusion that content was not discoverable because the path to it wasn’t in the link list begs the question, does all content have to be discoverable via a screen reader link list? Should we, for example, start coding all buttons as links?
The facts presented do not support the conclusion. Not even the greenest of screen reader users would read the link list without also looking at content on the page. If the subjects read the page, and if the tabs were well-placed, then simply telling the screen reader to recognize an element as a link instead of a tab is unlikely to change its discoverability.
The conclusion suggest research subjects were somehow led to believe the link list should be part of the path. Maybe it was something about the way the task was described or something the researcher said. Perhaps a leading word like “page” was used. And, it may be possible that a user expects all pages to be reachable via a link. Tabs are not links to other pages, or to other elements on the same page.
The second screen reader finding was stated as follows:
“The arrow key shortcuts also impact screen reader users. These users have to use a pass through modifier key (ie. the alt key) to activate each tab, which bypasses the native screen reader arrow key behaviour (reading line by line, word by word, or character by character) and instead passes the keystroke directly through to the page.”
First, none of us have heard of a screen reader that use alt as a pass-through key. We can hope that is either an editorial oversight or that we will get to learn about another screen reader!
More to the point, it is not necessary to use a pass-through key to activate a tab with any screen reader, or to read any part of a tab (like its name for example). In fact, it should not be necessary to use a pass-through key to operate any ARIA widget with any recent version of a screen reader. Tabs, in particular, can be accessed in either the reading or interaction mode of screen readers that have such modes. This finding suggest that either the screen reader used in the usability test was several years out of date or that there was something wrong with the implementation.
Are we saying there is no value in these findings? Definitely not. Clearly subjects struggled and the tabs were part of those struggles. There were some subjects whose expectations were not met.
Don’t user expectations trump efficiency in good design?
Designers love to innovate, but it does not always lead to a more enjoyable experience. It is not hard to find articles from reputable designers cautioning against change for the sake of efficiency. For example, here is an excellent article about why you should not prioritize efficiency over expectations.
So, if usability testing uncovers expectations that all tab elements should be in the page’s tab sequence, isn’t putting them in the tab sequence more important than adopting a new approach for the sake of efficiency gains? Similarly, if testing with screen reader users finds that they expect to locate all content via a list of links, wouldn’t it be best to make that possible?
We think there is a response to these specific expectations that is much more beneficial to all users than recoding the interface to match them. That is because:
1. Forcing a tab to appear in a link list means leaving out the ARIA semantics.
2. Not telling assistive technologies the UI contains tabs disables all assistive technology intervention. That means one user might experience a questionable benefit but all assistive technologies users will lose clear benefits, including the benefits to screen reader users we described above.
3. To facilitate both ease of use and efficiency, desktop GUIs do not put each tab in the tab sequence. A goal of ARIA is to provide first-class accessibility on the web by adopting the best of what the desktop world offers. This approach scales well because of the large number of people who experience desktop GUIs in their browsers and in the operating systems that run those browsers.
4. We cannot get the advantages of desktop GUIs on the web if we do not aggressively work to change user expectations from the clumsy and inefficient web 1.0 paradigm to the desktop paradigm.
Given we have a real need for reshaping user expectations, we should return to Aurora Bedford’s article on the balance between efficiency and expectations and consider the advice it gives us. That is, what we need to do as accessibility professionals and web developers to move web 2.0 accessibility forward is learn how to best help users master this new world as well as they have mastered the old one.
In other words, the most beneficial response to the above findings is to dig a little deeper so you can find ARIA-compatible ways of helping the user. It is imperative we conduct such research so people with disabilities do not fall further behind.
If I can’t do that research, isn’t it safer to avoid ARIA?
If your sites have GUI elements, we strongly encourage you to push forward with ARIA 1.0. The support for it is now robust enough that users will not experience actual roadblocks that you could blame on ARIA implementation if it is done well. We have some more information about doing it well further down.
But, if you are concerned by the argument that screen reader users do not expect GUIs on the web so cannot deal with them in that context, we believe such fear is overblown and that the argument itself is unreasonable. While there is clearly evidence that some users experience surprise, we have not seen evidence that the element of surprise completely disables them.
First, remember that there is a high probability these same users have learned to use tabs on the desktop or on a mobile device. If they have not already had that experience, other users just like them have.
Let’s look at the idea of getting surprised by finding tabs out of context more closely. Imagine you find a door in a solid rock face. Not a place you’d typically expect to find a normal door, but here’s what you’d probably do if you did: you’d reach out and grasp the door handle, then open the door by pulling it towards you. If that didn’t work, you’d try to rotate or push the door handle, then pull the door towards you.
The interesting question is why you’d do those things. The answer is twofold: firstly because you’d draw on your previous experience of doors, and secondly because the door itself encourages you toward certain actions.
The proper term for this is “affordances”. It was introduced into Human-Computer Interaction by Donald Norman in 1988. Here’s what Norman says about affordances:
“…the term affordance refers to the perceived and actual properties of the thing, primarily those fundamental properties that determine just how the thing could possibly be used.”
A set of tabs has visual affordance because they’re usually styled to look like those old-school rolodex dividers. People recognise these visual traits, and draw on their previous experiences of using tabs (in the real or digital worlds) in order to use them on the web.
Visual affordance is no help if you can’t see of course, but ARIA provides an equivalent set of affordances for screen reader users. People hear there is a set of tabs, and draw on their previous experiences of using tabs (in the real or digital worlds) in order to use them on the web.
Using ARIA
When you use HTML you get a lot of accessibility for free. The browser tells assistive technologies such as screen readers what each element does, what state it’s in, and how to refer to it. This is why it’s a good idea to use the appropriate HTML element for the job when it exists. You can find out more about the mechanics that underpin this by reading Accessibility APIs: a key to web accessibility.
When you add ARIA to HTML it changes the accessibility information the browser exposes to assistive technologies like screen readers. Understanding this relationship will help you use ARIA effectively, and we recommend reading the accessibility tree: a training guide for advanced web development.
One of the challenges of using ARIA is that it’s invisible. Unless you use a screen reader, there is no way to tell whether ARIA has been used or whether it has been used correctly. The Visual ARIA tool helps developers overcome this problem, by visualizing ARIA roles and attributes on the page.
As a young standard ARIA has not yet been fully integrated into many development and testing tools. But that is changing. Open source accessibility checkers such as aXe or Tenon include robust ARIA semantic checks.
Incorrectly coded ARIA is bad for accessibility. That is nothing unique to ARIA, it applies to any standard and code. In fact the author of the original article used the aria-selected attribute incorrectly (because aria-selected cannot be used on a link). But when used correctly ARIA makes your web content sing.
Final thoughts
We mentioned at the start of this article that we are four competent screen reader users with a huge interest in ARIA. We are also “real users”, which means we are no strangers to struggle. We encounter access barriers on a daily basis.
For us, a big part of why we work on accessibility is that we don’t want others to have to experience those struggles and be held back instead of being raised up by technology. Collectively, we have spent countless personal and professional hours both observing and helping other people with a variety of disabilities, of all ages, and every skill level use technology. This is all to say that we believe we have a strong understanding of both sides of this coin — the user side and the engineering side. We live and breathe both. And, while some will be quick to point out that we are power users of our assistive technologies, we are not focused on power users. If there is one thing we are vigilant about, it is staying in touch with people of all stripes.
We also do not believe ARIA is the perfect “be all, end all” solution. It is a work in progress, and while we are strong advocates, we are also among its most prolific critics. We are anxious for more people to join in the process of questioning assumptions, generating and testing ideas, and making it possible for others to benefit from those ideas.
As you work to make the web more accessible, please remember ..:
1. If you encounter difficulties putting ARIA to work, don’t automatically assume ARIA is the problem and toss it out. Let’s work together to make it work. It is the best shot we have at creating an inclusive web.
2. When conducting research studies with people with disabilities, be extra careful about making generalizations. Use your studies not only to help make a specific sight better but also to constructively inform how we employ accessibility technologies.
Finally, if your site includes GUI components, please use ARIA. ARIA 1.1 will have a very robust authoring practices guide, including reference implementations.
Further Background Reading
Disability Accommodation Cost Guides
Thriving in Trade School with a Disability
Discrimination And Addiction: How To Overcome Prejudice Without Relying On Drugs Or Alcohol
Accessibility and Employment: What People with Disabilities Need to Know
Wheelchair and Handicap Ramp Cost Guide
Dating When Blind or Visually Impaired — From Single and Ready to Mingle to Off the Market
Disaster Safety for People with Disabilities: What to Do When Emergency Weather Strikes
