Four Things We Learnt From Facilitating Usability Testing Sessions With Blind Users

Vedran Arnautovic
Apr 11, 2017 · 4 min read

Recently two of my colleagues (Todd Moore, Stephanie Moss) and I tested parts of SEEK’s site with vision impaired users.

It was my first time facilitating usability sessions with users who entirely rely on screen readers to navigate the web.

We learnt a lot of specific things about our site — things that worked and things we needed to improve.

However, there were four things we learned which were not necessarily about specific issues with our site — these were lessons which will be useful the next time we run usability sessions with vision impaired users.

Hopefully, you will find them useful too.

1. Just because someone is blind, doesn’t mean they are an expert at using a screen reader

I made the wrong assumption that anyone who is blind is going to be an expert screen reader user. How else will they get around the internet, right?

Wrong.

Just like there are varying degrees of computer literacy and proficiency in sighted people, there are the same variations with people who can’t see. This sounds, and is, so obvious. However, I assumed that being more dependant on technology, like screen readers, would force blind users to master the technology.

When planning a usability session with blind users, allow more time. This is not to say that blind people are slower at navigating the web — the reality is that many websites are not optimised for navigation with screen readers. Mistakes happen, and those mistakes take longer to recover from when using a screen reader.

If you’re testing specific features you built for blind users, be prepared for them not to be discovered, and plan for how you will handle that in your session.

For example, we implemented ARIA Regions on our site, only to find that none of the participants we tested with used them.

2. Being blind may affect people’s propensity to learn new tools and technologies

Learning new tools and technologies can be hard (I recently gave up on trying to figure out how to use Snapchat — true story). Now imagine doing it without sight. It’s harder, right? It’s harder because most online tools and sites are not designed with blind people in mind.

One of the things we were keen to learn was what tools our participants used. We expected that they would switch between different screen readers, devices and browsers, depending on the task, location and context.

What we found was that they all stuck with a combination of tools they were familiar with. When questioned, they quoted high effort associated with learning new tools as a deterrent. This finding may differ depending on your user base and sample size.

If you need to test different browsers, screen readers and devices, consider using a survey to pre-qualify your participants in detail.

3. Getting comfortable facilitating with blind users

To be honest, I was very nervous before running these sessions. I had no idea if the script I prepared was up to scratch. I found it really difficult to gauge upfront how long we would need to complete the tasks.

Most of all, I was worried I would say something insensitive, something that may offend the participants. I tried hard not to use phrases like ‘tell me what you see’ or ‘tell me what you are looking for right now?’. Instead, I used phrases like ‘tell me what’s happening now’ or ‘tell me what you’re searching for’.

It turned out I was stressing for no reason. The participants regularly used phrases like ‘I’m looking for…’ or ‘I can see that x is not working…’

My advice would be: just relax. Really listen to your participants, hear how they speak and echo the language they are using.

4. Blind users are amazing at creating mental models of websites

One of the things that really struck us is to what level of detail our participants described the structure and the content of the pages they visited — they formed detailed mental models of the pages they interacted with.

The descriptions didn’t always match how the site was actually laid out, but that made them even more interesting — they gave us an insight into how our site was perceived by blind users.

This is an opportunity to learn about how you should structure the content of your site and pages for blind users. Next time we run these sessions, I will definitely include an activity where we get participants to describe the structure of the pages they are interacting with.

It’s important to note that mental models of sighted users often don’t match the system model either — this is where we have a gulf of evaluation. Understanding and reducing the size of the gulf of evaluation helps us design better experiences, not just for screen reader users, but for all users.

This reminds me of the amazing artwork by Alby Letoy who was commissioned by TravelSupermarket.com to produce illustrations of travel destinations, based on descriptions provided by visually impaired people who visited them.

Here is one of those illustrations

Image for post
Image for post
Image source: http://www.lonelyplanet.com/news/2016/11/09/illustrations-alby-letoy-visual-impairment/

Like with anything, practice makes perfect. I’m looking forward to applying these lesssons the next time we run usability testing sessions with blind users.

Big thanks to Kayla Heffernan for a detailed review of this post.

SEEK blog

At SEEK we’ve created a community of valued, talented…

Thanks to Rob Scherer and Kayla J Heffernan

Vedran Arnautovic

Written by

Designing in Australia, assembled in Bosnia. Product Design Manager at Zendesk. www.vedran.io

SEEK blog

SEEK blog

At SEEK we’ve created a community of valued, talented, diverse individuals that really know their stuff. Enjoy our Product & Technical insights…

Vedran Arnautovic

Written by

Designing in Australia, assembled in Bosnia. Product Design Manager at Zendesk. www.vedran.io

SEEK blog

SEEK blog

At SEEK we’ve created a community of valued, talented, diverse individuals that really know their stuff. Enjoy our Product & Technical insights…

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store