Which accessibility testing tool should you use?

Paul Stanton
May 17, 2018 · 11 min read

Starting our test

I needed a real user interface to test, something fairly small with a handful of form elements and interactions, so I started where all our Continuum users start, our sign-in screen. There’s a small amount of interaction design here, users will move between a few different forms in-place but there’s not an awful lot of markup involved, so for the purposes of this post it’s a good example of how we can work through the various issues flagged by the tools.

Our sign-in UI, about to be put through its paces

WAVE Web Accessibility Evaluation Tool

This is usually the first browser extension developers think of when accessibility testing comes up, but as I’ll show in the post, it’s probably not the first one you should reach for.

Inspecting the red icon in the UI leads me to the related element (the `.signin-brand` image, missing an `alt` attribute

Lighthouse

If you’re running an up-to date version of Chrome, you probably already have Lighthouse because it’s built right in! Open devtools and go to the ‘Audits’ tab, hit the ‘Perform an Audit’ button and you’re given a list of the audits that Lighthouse can perform, we’re only going to run the accessibility audit to save time.

aXe Browser Extension

I’m a big, big fan of aXe, the extension adds a new tab to Chrome’s Dev Tools with a big blue ‘analyze’ button, once you hit that you’re shown a very nice list of issues (I’ve filtered to just show violations here, but there’s also a list of other things to review) with really useful related information.

The highlight button does a much better job of showing the element related to the violation

WCAG Accessibility Audit Developer UI

It only checked four things, and passed them all. In the bin with you.

I uninstalled this one.

SiteImprove Accessibility Checker

SiteImprove is very popular in the waters in which we sail (local & central government, higher education, not-for-profit etc…) so it’s very useful to have a SiteImprove tool to verify the accessibility of our UIs.

Very similar results to aXe, although that aria-atomic is a new one! (review issues are filtered out)

Tenon.io

Tenon is different as it’s a web service you can use much like the W3C HTML Validator we all know and love, but for accessibility. Simply give it a link or paste in the markup of your UI and it’ll generate a report for you. There‘s multiple (paid) ways to integrate Tenon with your build tooling or CI servers, but that’s meat for another blog post.

Pretty much the same results as the others, nicely presented with markup examples.

The scores on the accessible doors?

Barring one shambolic effort, all of the tools gave fairly consistent results for this—admittedly limited—UI test. For me to give them some form of ranking it came down to how the information is presented and what tools it gives me to resolve the issues it raises. Realistically there’s no silver bullet and we’ll likely use most if not all of these tools to sanity check each UI, but the first tool I use in nearly all cases is the aXe browser extension in Chrome. The information is clear, well organised and the highlighting and dev tools integration make it the best tool out of the ones I’ve tested so far. I’m also very interested in integrating aXe core or Tenon.io into our CI process in the near future, for more automated on-going testing.

  1. SiteImprove
  2. Tenon
  3. WAVE
  4. Lighthouse*

The fixes

So I know I have things to fix, and to be honest I’m not going to bore you with those, I’m only going to use aXe during my initial pass and then see what the other tools tell me.

  • Add <main> wrapper around the content to act as an aria-landmark region
  • Change the ‘Pulsar’ text into a h1 heading (with CSS fixes to maintain size)
  • Remove a bit of javascript which added incrementing tabindex values (1, 2, 3 etc) in favour of simply 0 or -1
  • Added id attributes to the ‘forgot password’ and ‘sign in’ forms to act as the target for the related skip-links

aXe = No violations 🎉

WAVE = no errors, no alerts 🎉

Lighthouse = no errors 🎉

SiteImprove 💥

OK, so SiteImprove still isn’t happy…

The ‘Enter your username and password’ alert is an element with role=”alert”

Tenon.io 💥

Lastly, a few more nitpicky bits from Tenon…

Findings

After experimenting with this range of browser based accessibility testing tools, the most important thing to take away is that no one tool gave me a complete list of issues found by the others, you really do have to test in multiple tools.

  1. SiteImprove
  2. Tenon
  3. WAVE
  4. Lighthouse

Pulsar

The product design team at Jadu

Thanks to Nikos Vasileiadis.

Paul Stanton

Written by

I design things, I build things. — UI/UX person @Jadu

Pulsar

Pulsar

The product design team at Jadu