The ‘work around the work’ of testing the accessibility conformance of digital products

In anticipation of Global Accessibility Awareness Day (GAAD) on 16 May 2024 and my talk on accessibility auditing at UX Scotland later this month, I’ve put together a list of things to consider when testing (auditing) digital products for accessibility conformance, focusing on using the Voluntary Product Accessibility Template (VPAT®) in order to discover, report, and learn about accessibility conformance.

I’ve broken the list down into a simple timeline:

• Before — preparation you can do individually and as time allows
• During — working with the product you’re testing and recording findings
• After — sharing findings and collaborating with your product team
• Beyond — using your audit and findings to help inform a broader accessibility strategy in your organization

The full list is below with some helpful links for more information, but first here’s a single-page break down:

BEFORE — as an individual

Learn

• Learn about digital accessibility (e.g. W3C course)
• Assistive technology (AT) and the accessibility tree (e.g. web dev)
• Understand how people with disabilities use Ats (e.g. WAI fundamentals)
• Be comfortable with all the WCAG criteria (WAI Understanding)
• Differences between accessibility audit types (Deque)
• What consumers need from an audit
• About the VPAT in particular (LevelAccess)
• Existing policy inc. legal

Tech

• Choose screen reader or other assistive tech etc (AbilityNet)
• Learn to use screen reader on relevant platform (RNIB)
• Find keyboard or gesture cheat sheet (Deque screenreader resources)
• Remember key shortcuts and exit mechanisms (e.g. ‘Ctrl’)
• Get comfortable exploring varied content
• Round up a library of automated tools, checkers, and validators (WAI overview of tools)

Prepare

• Choose and download the required VPAT edition (ITI)
• Read the VPAT intro
• Decide when to audit
• Understand the scope of your auditable content (inc docs)
• Define relevant workflows if necessary
• Choose level of testing, A, AA, AAA
• Prepare working document
• Plan work time

DURING — with the product

Inspect

• Read criterion description
• Understand requirements, exceptions, techniques, and example failures
• Read notes and tips from prior audits
• Determine which (semi) automated testing tools to use
• Check content manually and using automated tools
• Inspect content using dev tools and accessibility tree

Record

• Note version and date of testing
• Describe issues found
• Who is affected and why it’s a problem
• Capture screenshots
• Anything else to check later
• Suggestions for improvement or best practice
• Frequency and severity of issues
• Prepare statement
• Improve guidance notes for the next audit

VPAT

• Fill in product information
• Describe evaluation methods and tools
• Describe what content was tested and to what level
• Enter each conformance level (Supports, etc)
• Fill in conformance statements for each criteria
• Double-check / review statements and conformance level
• Check whether a legal disclaimer is required

AFTER — with the team

Analyze

• Number and frequency of failures
• Assess severity (e.g. NHS Checklist includes severity)
• Determine level of effort to remediate
• Unexpectedness of failure
• Compare with other VPATs (e.g. Google VPATS)
• Prioritize issues

Report

• Write up a summary
• Report back to the team
• Share documentation and process
• Determine strategy for tickets
• Create bug tickets
• Demo trickier issues to developers
• Share or publish final VPAT

Retrospective

• What issues occurred frequently and why
• What could have been caught earlier
• What processes can we improve (team and audit)
• What checklists could developers use
• How can we raise awareness
• What can we share with other teams

BEYOND — with the organization

Advocate

• Act as advocate (not the police)
• Check status of tickets occasionally (e.g. fixed in passing)
• Push for remediation
• Spot check of fixes
• Monitor new content
• Highlight issues
• Validation audit

Expand

• Broader awareness
• Early accessibility adoption on new projects
• Product coverage
• Product portfolio review and status
• Impact assessment
• Developer training and support
• Other types of conformance statements

Maturity

• Establish maturity model, levels, and current state (e.g. W3C or AbilityNet)
• Team, dept, and org policies (e.g. WAI planning and policies)
• Periodic accessibility checks to ensure continued compliance
• Accessibility testing at all stages of dev lifecycle (e.g. Deque types of accessibility testing)
• Inclusive user testing (e.g. WAI involving users)
• Accessible by default (e.g. Gov Food Standards Agency)
• Adapt with changing standards (e.g. European Accessibility Act ) and laws (e.g. A11y Quest list of laws)