Web Accessibility Testing

Overview

Who wants to increase their website’s traffic and potential usership by more than 25%? (…Everyone raises their hands…) Splendid! Adding accessibility to your website is a great way to do it, and in many cases it’s mandated by law.

In this article, I’ll discuss some common accessibility solutions and how to implement and maintain them on your website.

Additionally, I’ll share how we at WalkMe automate the maintenance of our accessibility code and ensure it doesn’t break our app.

The power of the Web is in its universality. Access by everyone regardless of disability is an essential aspect.

(Tim Berners-Lee, W3C Director and inventor of the World Wide Web)

What is website accessibility?

Web accessibility refers to the inclusive practice of removing barriers that prevent interaction with, or access to websites, by people with disabilities. When sites are correctly designed, developed and edited, all users have equal access to information and functionality (Wikipedia).

Ways to enhance website accessibility

There are many ways to enhance a website’s accessibility, including the following:

• Adding meaningful textual HTML tags (e.g., title, alt, aria-label, etc.) — This helps blind users who are using text-to-speech software.
• Marking links as instead of merely coloring them — This ensures that color-blind users will be able to distinguish them from ordinary text.
• Coding pages so that users can navigate by keyboard alone — This helps users who cannot use a mouse or even a standard keyboard.

Enhancing your website’s accessibility can be quite complex; while the basics of accessibility, such as tags and attributes, are easy enough to implement, the process of making a website accessible without hindering its functionality requires more than just tweaking HTML.

WalkMe is no stranger to this topic; as a 3rd party library, we must match the accessibility standards of our hosting websites. In doing so, we must preserve not only our own app’s functionality, but also our hosts’.

Overwhelmed? Don’t be! In this post, I’ll explain how we use tests to automate the quality assurance of our accessibility code, and how you can too.

How to avoid complications when enhancing your website’s accessibility

Use unit testing

Traditional unit testing with Jest gives us great coverage; we can test that our accessibility attributes are set correctly and that we don’t hinder functionality while adding to and altering HTML.

Here is an example of a unit testing suite that tests that a row element and close button receive the correct attributes (tabIndex, role, etc.):

Example of unit testing for accessibility attributes

And here is the output of our unit test:

Output summary for Jest testing run

Unit testing give us a net of confidence of coverage. But is it enough?

Since unit tests are written for specific functions, there are many cases and scenarios they don’t cover, such as element focus, in-app behavior not relegated to a singular unit, and more complex scenarios involving keyboard interactions. As such, we need to bring in some additional tools. Enter Puppeteer and aXe!

1) Use Puppeteer

If you read my previous post, you probably noticed that we love using Puppeteer.

Puppeteer is a Node library that provides a high-level API to control headless Chrome or Chromium over the DevTools Protocol. It can also be configured to use full (non-headless) Chrome or Chromium.

Here are some common accessibility tests to perform using Puppeteer:

Keyboard functionality testing using Puppeteer

On an accessible website, when an element has keyboard focus, users should be able activate or manipulate it with the keyboard alone. Keyboard accessibility is one of the most important aspects of web accessibility. Testing with a keyboard is an essential part of any accessibility evaluation.

The basics of keyboard testing are simple: The tab key must navigate through interactive elements — links, buttons, fields for inputting text, etc., and enter and spacebar must select or toggle elements.

Tip: You are not just testing that users can interact with the page using the keyboard; you also want to ensure that all interactions are predictable.

View state toggle test:

To test that the spacebar key toggles view state, we will write a test that includes the following steps:

1. Navigate to a category.
2. Test accessibility properties (tabIndex, role, aria-expanded, etc.).
3. Press the spacebar key; it should toggle the property aria-expanded from true to false, and vice versa:

Expand and collapse category with spacebar key

And here is the code for the test that checks this scenario:

Expand and collapse testing

In the above code, we first fetch the category element, and then we check that category has tabIndex=0, role=button, and focus. We accomplish this using Puppeteer’s page.evaluate and getAttribute methods.

In addition, we check that the attribute: aria-expanded is false when collapsed.

After validating these properties, we open the category by pressing the spacebar key or invoking page.keyboard.press(<keyName>), and further validate that the attribute: aria-expanded is true now that the category is opened.

Keyboard focus and navigation order test:

To test keyboard focus and navigation order, we’ll run a test ensuring that pressing on tab or arrow-down keys will move the keyboard focus to the next interactive element in view:

Navigate elements inside containers with tab and arrow-down keys

Here is the code of the test that checks this scenario:

Tab navigation UI testing

In the above code, we first fetch all row elements, and then we check that each row has tabIndex=0, role=button and that the row is focused. We achieve this using Puppeteer’s page.evaluate and getAttribute methods.

After validating those properties, we move to the next element by “pressing” the tab key or arrow-down key with page.keyboard.press(<keyName>).

Accessibility behavior testing using Puppeteer

Sometimes, when using accessibility on dynamic or interactive pages, we need to dynamically change the behavior of elements in a page to cover all our bases.

For example:

• Tab-panel—When interacting with tab-panels, clicking the arrow-right and arrow-left keys should switch between panels.
• Focus selector—When there is an interaction with a new container element, we need to decide which of the container’s inner elements will receive focus first.
• Lock navigation—When inside a dialog, we need to ensure that elements outside the dialog don’t receive focus.

Here’s a concise example of accessible behavior in tab-panel navigation:

1. In the beginning, the selected tab (Tasks, in the GIF below) should be focused with tabIndex="0" and with aria-selected="true".
2. The other tabs should have tabIndex="-1" (unreachable), and aria-selected="false".
3. when pressing the arrow-right key, the focus should move to the next tab and the attributes should switch:

Switch tabs using arrow keys

And here is the code for the test that checks this scenario:

Tab-panel keyboard interaction testing

In the above code, we first fetch the tab buttons’ elements, and then we check that the selected tab has tabIndex=0, role=tab, aria-selected=true, and focus.

We do this, again, using Puppeteer’s page.evaluate and getAttribute methods. The rest of the buttons should have aria-selected=false and tabIndex=-1 (unreachable).

After validating these properties, navigate to the next tab by pressing the arrow-right key or invoking page.keyboard.press(<keyName>), and validate that the previous tab has aria-selected=false and tabIndex=-1, and that the current tab hastabIndex=0, aria-selected=true, and focus.

2) Use aXe

aXe is an open source JavaScript library for accessibility testing.

The aXe API provides the following benefits:

• It runs in any modern browser.
• It runs locally; no connection to a third-party server is necessary.
• It performs violation-checking on multiple levels of nested iFrames.
• It provides a list of rules/tests, ensuring rules have been run against the entire document.
• It is designed to work with existing testing infrastructures.

How to use aXe

First, inject aXe-core into the tested page, and then execute axe.run():

axe.run(context, options, callback);

• context — (Optional) Defines the scope of the analysis — the part of the DOM that you would like to analyze. This will typically be the document object or a specific selector such as class name, ID, selector, etc.
• options — (Optional) Set of options passed into rules or checks, temporarily modifying them. This contrasts with axe.configure, which is more permanent. Click here for more information about the configuration.
• callback — (Optional) The callback function, which receives either null or an error result as the first parameter, and the results object, when analysis is completed successfully, or undefined, if it is not, as the second parameter.

What do aXe results contain?

The aXe results contain a set of rules, which are actually tests, that run on the page and check accessibility metrics, such as color-contrast, duplicate-id, aria attributes, and more. The evaluated test results are divided to the following groups:

• Passes — These results indicate which elements satisfied the rules.
• Violations — These results indicate which elements didn’t satisfy the rules.
• Inapplicable — These results indicate which rules did not run because no matching content was found on the page. For example, with no video, these rules won’t run.
• Incomplete — These results indicate the run was aborted and further testing is required. This can happen either because of technical restrictions as to what the rules can test, or because a JavaScript error occurred.

Each object returned in the following arrays has information about the test, such as description, tested rule, tested DOM elements, tags, impact, etc.

Example of the returned result object

Using aXe results to enhance test output

Usually we will search for the violations property.

In addition, there are four levels of impact: “minor”, “moderate”, “serious”, or “critical”. The following code will only search for violations with “serious” and “critical” impact.

How to use the aXe tool with Puppeteer to get violations

And here is the output of the test:

Example of a failure test output. The output contains the error types and the problematic elements

What can’t we test?

• Unfortunately, Puppeteer is supported on Chrome browsers only. As we know, different browsers can behave differently and provide unexpected results (for example, different focus, element selectors, CSS styles, etc.), so we may need to incorporate different tools for a full battery of cross-browser testing.
• Screen readers are frequently used by differently-abled individuals, and this has implications for our attributes and functions. Different screen readers can work differently not only from each other, but from browser to browser (e.g., JAWS overrides the arrow-key functionality). As such, it’s difficult to test all screen readers in all scenarios. In order to ameliorate this problem, try to run tests with screen readers turned on and check whether you still get the expected behavior.

Summary

In this post, I gave a brief intro to web accessibility concepts and explained what makes a website accessible. I mentioned some principals of web accessibility and the challenges we face when implementing them.

I also showed examples of the following methods for automatically testing accessibility during code changes:

• Unit testing.
• UI testing with Puppeteer.
• Testing for accessibility violations using the aXe library.

I hope this article opened your eyes to a new front-end niche. As should be evident by now, while implementing accessibility, unexpected behaviors will probably manifest, and you need to be creative to overcome them. I welcome you to comment below and share any problems you’ve faced in implementing accessibility, and the creative ways you solved them.