How to run a thousand accessibility tests

An open-source project from CVS Health combines 11 web accessibility testing tools into one automated super-tool. Jonathan Pool is a staff software development engineer in the CVS Health Web Core team.

A testing paradox

It’s unanimous: Your website should be accessible. That helps users of all ages and abilities enjoy it in both simple and complicated situations. By following inclusive web standards and practices, you also limit your legal, economic, and reputational risks.

To know whether your website is accessible, you test it, but this can be slow, difficult, and costly. The best testing includes people with various disabilities using the site with various computers, mobile devices, browsers, and special-purpose applications.

To make testing faster and cheaper, we can automate parts of it. Many tools now discover and document accessibility faults. As CVS Health research shows, each tool discovers problems missed by other tools, so the most effective test automation, ensemble testing, leverages multiple tools.

A paradox: Ensemble testing, though effective, is rarely used. Few websites are tested with more than one tool. Why? Because it is tedious to procure, install, configure, and run multiple tools, then collate their reports in various formats and vocabularies.

A solution

CVS Health is developing a solution to this problem: a pair of applications named “Testaro” and “Testilo”. Together, they dramatically boost the feasibility of ensemble testing. In their current versions, they run eleven different web accessibility testing tools, each performing an average of about 90 tests, for a total of about 1,000 tests, and report results within about three minutes to both technical and nontechnical audiences.

One website owner recently told us: “I am … dismayed at the things I missed. … Thank you so much for the work that went into catching the errors.”

These tools are free for everybody to use. They are among the growing collection of CVS Health’s open-source software repositories on GitHub.

Here are the eleven tools leveraged by Testaro and Testilo:

1. Alfa, from Siteimprove (Denmark)
2. ASLint, from eSSENTIAL Accessibility (Canada)
3. Axe, from Deque (USA)
4. Editoria11y, from Princeton University (USA)
5. Equal Access, from IBM (USA)
6. HTML CodeSniffer, from Squiz Labs (Australia)
7. Nu Html Checker, from the World Wide Web Consortium (USA)
8. QualWeb, from the University of Lisbon (Portugal)
9. Testaro, from CVS Health (USA)
10. WallyAX, from Wally (USA)
11. WAVE, from Utah State University (USA)

Impact

What will happen if ensemble accessibility testing is widely adopted? Here are some predictions:

• More accessibility defects will be discovered automatically. Web developers who rely on automated checking will catch and correct more problems.
• Test engineers will focus more on subtle problems requiring human judgment—the problems they most enjoy working on.
• Website revisions that accidentally harm accessibility will be caught earlier through routine automated monitoring.
• Tools will cooperate more and compete less. The “Use our tool!” refrain will become less seductive. Tool makers will be more at liberty to specialize, instead of trying to cover the entire accessibility terrain.
• Tool accountability will improve. Tool disagreements will become easier to document. Users will more easily notify tool makers and ask them to resolve their discrepancies.
• Commercial accessibility businesses will leverage open-source ensemble testing to help organizations better manage complex activities such as human testing, defect prevention, design standardization, accountability, and risk estimation.

What you need

You can make use of Testaro and Testilo on a single computer if you can install software and use a terminal interface. The README files in both repositories explain how.

An organization wanting to integrate Testaro and Testilo into its automated testing workflows can call them programmatically.

It is also possible to create user interfaces to Testaro and Testilo. A web-based application supervising the testing process (let’s call it the “controller”) will be responsible for request, test, and report management, but can rely on Testaro and Testilo for test configuration, performance, and reporting.

How they work

Here is how Testaro and Testilo would collaborate in a typical user-facing service:

• The controller (running on a web server) displays a web form giving the user the testing options allowed by the organization’s policies.
• The user fills out and submits the form.
• The controller uses Testilo to prepare a Testaro job according to the options submitted by the user.
• Testaro is continuously running on one or more consumer-type Macintosh, Windows, or Linux workstations (testing “agents”).
• Each Testaro agent periodically asks the controller for a job.
• Now that a job is ready, the controller assigns it to a requesting agent.
• Testaro on that agent performs the job by running the tests that the job specifies.
• When the job is complete, Testaro sends its report back to the controller.
• The controller uses Testilo to convert the Testaro report to a human-friendly report satisfying the user’s requirements.
• The controller stores the report in a file.
• The controller sends a notice to the user containing a link to the report.
• The user follows the link and retrieves the report.

Thus, Testaro and Testilo have complementary purposes and operate on different kinds of hardware. Testaro does the testing and operates on a consumer-type workstation. Testilo is a utility that prepares jobs for Testaro and enhances reports from Testaro, used by a controller running on a server.

The organization’s policies determine who may request what kinds of testing, the retention periods for reports, and other business rules. For example, some users may be allowed to test a single page at one time, while other users may be authorized to request repetitive periodic testing of multiple pages for surveillance of accessibility health. Neither Testaro nor Testilo tries to anticipate the diversity of such policies, so it is your controller that must implement those.

Capabilities and limits

Testilo makes it easy to give instructions to Testaro. With a few parameters, Testilo can assemble a job in the syntax that Testaro requires.

A job can make the Testaro tools run all their tests or only particular ones. It can also make the tools adopt the viewport and touch features of any of 125 devices, including computers, tablets, and telephones.

Testaro analyzes the reports from the eleven tools to produce, for each tool, a “standardized” report. This converts the tool’s distinct report to a uniform format. Testaro then combines the standardized reports into one document.

Testilo analyzes that document and organizes the resulting information for efficient use. If the requester wants high-level management insight, Testilo can aggregate the results into total scores and report score comparisons and/or score changes over time, with tables, bar charts, and line graphs. If the requester wants detailed diagnoses for designers and developers, Testilo can organize the data by problem type or by HTML element. If the requester wants a rapid pointer to a single high-priority defect to work on, Testilo can prepare a job that performs particular tests and can then select a defect for action.

These and other existing capabilities allow Testaro and Testilo to be put to practical use, but it is still fair to call them prototypes rather than mature products. There is more to do. New tests can be created to extend the scope of Testaro, particularly tests of “transactional accessibility” that emulate a website user performing a complete task. With further research, it may become possible to test with assistive software running, to test with robust device emulators and physical mobile devices, and to test embedded documents and native mobile applications. Testilo can be equipped with better logic to identify duplication and disagreement among tools and to evaluate the credibility of each of the thousand tests. New tools may become available for integration into Testaro and Testilo.

Want to help?

Although Testaro and Testilo are sponsored by CVS Health, developers anywhere can help make them more powerful. And anybody who uses these applications can help by reporting errors and requesting new features. The README files describe the process for contributing to the code, and the GitHub repositories allow you to post comments and suggestions as “issues”. We look forward to welcoming you as a collaborator, a user, or both.

© 2024 CVS Health and/or one of its affiliates. All rights reserved.