How to run accessibility testing demos

This is the sixth post in a series on accessibility from Shopify’s UX team, published biweekly. Check out the introduction here.

The need for accessibility is substantial: according to surveys, 22% of the population in USA, 13.7% in Canada, and 4% in Mexico have some kind of impairment, which roughly translates to one in every eight people in North America with some sort of disability.

As a front-end developer at Shopify, I’ve had the opportunity to create accessible interfaces, and to test those interfaces both through assistive technology and with users who have impairments. As I dug into my research on accessibility one quote resonated deeply with me, shaping my perspective:

“…disability is a conflict between someone’s functional capability and the world we have constructed.”
Sarah Horton & Whitney Quesenbery — A Web for Everyone

I found it eye-opening to move away from thinking of disabilities as bound to people, and instead place the emphasis on the things used by people. It became obvious that removing this conflict requires us to re-construct the world in a way that works for everyone.

“It’s worth remembering that accessibility is often about empathy, and computers are not inclined to be empathetic: only humans instructing those machines are capable of that.”
Heydon Pickering — Apps For All

Having empathy for users with impairment is critical. We try to understand different types of impairments, such as limitations on vision, movement and/or cognition. We think about how each of those impairments affects the lives of millions as they use the multitude of products available online, like apps and websites.

But beyond empathy, what’s important is including real users in the conversation. As a result, we frequently invite users to test our products, not only to challenge our assumptions but also to teach us about the greater picture of using assistive technologies on the web. Such tests help deepen our understanding, replace assumptions for insights, and ensure the work we do is creating the impact we want.

“Find people who are fairly experienced using products like yours. If people use assistive technologies with your product, you probably want people who are skilled with their assistive technology. Later in testing, you might want to include some novices, but early on you want people who can teach you well.”
Shawn Henry, on testing — A web for everyone

Where are your personas?

Validating your work with personas is a well-known practice in many companies. People often tailor personas to represent their target audience in terms of interests, age, proficiency with various types of technology, objectives, etc. However, they often neglect impairments.

Enhancing existing personas or creating new ones with detailed impairments and struggles can help raise awareness and develop empathy. I can’t recommend enough reading the book “A Web for Everyone”, as it includes eight detailed personas that can truly inform the experiences you work on.

Testing FTW

Throughout the past two years, we’ve hosted four assistive technology tests sessions across our offices in Montréal, Ottawa, and Toronto. For each test, we had different scenarios, users, and scopes. We’ve learned a lot not just from the results of the tests themselves, but also on how to run such tests. We’d like to share how we’ve been hosting these test sessions, as well as lessons from both our mistakes and successes.

Aim at the right target

It can be tempting to try and test for everything, but it’s important to not overwhelm your user with too much information. Pinpoint what you want to test for, as information overload can otherwise lead to confusion and compromise your test results.

The first step is to have a set scope. During our latest session, held in Montreal last year, we decided to focus on our checkout process from the perspective of visually impaired users who use screen reading software and limited it to two hours. Through internal surveys, we’ve learned that tests running for over two hours may be too tiring for our participants.

Choose the right testers

For that session, we invited an active member of the visually impaired community highly skilled in using screen readers and other assistive technologies. In fact, he also teaches other users with visual impairments how to use them. As mentioned earlier, it’s important that you look for users who not only can test your product, but can also teach you and your company about the day-to-day life of using the web. Getting in touch with places like the Canadian Council of the Blind may help you find the right participants.

Start the experience off right

We’ve picked up a few pointers after running our test sessions. One thing often ignored is that the experience doesn’t just start with the test session itself, but begins as early as the invitation. We strive to use language that is inclusive, such as “we/us”, rather than the divisive “you/they”. In addition, we avoid problematic language such as “handicap” or “condition”.

We also strive to make our guests feel comfortable. It’s easy to take for granted daily habits like commuting. Make sure you give your guest user accessible directions (see this example on how to give directions to a visually-impaired person), and ask whether they have any special needs such as transportation preferences, or use a service dog. We also ask ahead of time if guests have any food allergies, or what we can do to help them move around and get settled in.

During the actual session, we’ve utilized three techniques that helped everyone get the most out of it:

  1. Employ storytelling techniques: when thinking about scenarios, try to come up with real world situations that your participant may find themselves in. For example: “You decided to buy six cupcakes for a friend and will be using a coupon you received the last time you made a purchase from this shop.” A story can make it easier for your participant to relate to the scenario due to possible past experiences, and can aid their understanding of the situation.
  2. Minimize potential frustration for your participant: it is possible that, due to the nature of the participant’s impairment, they may not be able to complete all the scenarios you’ve prepared. Even when a person knows they’re helping test a product, it can feel disappointing to get stuck. Of course, a large part of why we test our products for accessibility is to find out why this may occur, but we can still help minimize the frustration. Look into creating tasks that are well-scoped and achievable, and reassure the participant that they aren’t by any means failing: if anything, their inability to complete a task represents a problem on the product and not on them. Offer help when you feel they are stuck.
  3. Delegate the note-taking: if you’re facilitating the session yourself, let others do that for you and give your full dedicated attention to the participant. Find one or two people who can take note of any behaviors, questions, compliments or problems they find. The exception to this rule is if you can do it quickly enough that it won’t distract you or your guest from running the session.

Make some history

These test sessions are enlightening and we record them for future reference. By supplementing those recordings with the notes taken during the session, we’re able to create detailed issues. If anything is unclear, we can always revisit the session by watching the recording.

We’ve found it crucial to record all the audio (user comments while testing, proposed scenarios by the interviewer, etc), and also three video inputs: the participant’s face, their computer’s keyboard, and their screen.

  • User-facing camera: this allows us to evaluate facial expressions and reactions to the proposed scenarios and also to the product they’re using. This helps us feel more connected with what they are feeling and notice any points of struggle that may not have been verbalized.
  • Keyboard-facing camera: this gives us information on how users with visual impairments browse the web using their keyboards. It can also help showcase to attendees how the participant interacts with mobile touchscreen devices, as well as other assistive devices (such as bone conduction headphones, braille keyboards, etc).
  • Screen: perhaps the most important one, recording the user’s screen will show attendees how screen reading software works. It also emphasizes that sometimes, no matter how obvious some features are to users who don’t have visual impairments, they may be very troublesome to reach, use and understand for those who do. Having the browser occupying full screen may not be necessary for visually impaired users, so for the purposes of your recording, you should ask the user to either maximize the browser or help them with this task.

Physical set-up

Here’s how we setup one of our meeting rooms for people attending in person (the projected screen was being streamed to people who joined remotely):

User and keyboard-facing cameras, and external microphone connected to Host computer, streaming inputs via projector.
Example of content being presented to participants.

Educate

Your product is more than just design and code: it’s made by people, for people. A solid foundation on accessibility will allow you and your company to build a product that will embrace a wider audience, facilitating its usage worldwide.

Make this session educational and invite some people to attend it in person or online. Note that this depends on how comfortable your guest may be with the idea and their impairment: someone with a cognitive impairment such as attention deficit disorder may not feel as comfortable sharing their experience in front of an audience, for example.

Some of the amazing notes Kevin Clark took from the latest assistive tech demo.

You’ll want to go through your test scenarios first, so that the flow isn’t interrupted, but encourage attendees to take notes, and share their questions during the last 30 minutes of the session. A surprising amount of people who work with products on the web have never had a chance to see their work being used by someone with any sort of impairment and this will certainly be eye-opening and prompt all types of questions.

In our case, since we tested our product with a visually impaired user, silence was very important given that the user is relying on the sound output of the screen reading software. Try to assure that conversations among attendees (both physically or on the online streaming) are kept to a minimum so that these don’t interrupt the tests. A good tip is to mute the microphones of remote attendees joining the online stream.

Rinse and repeat

Constant questioning of your practices and running test sessions to get rid of any assumptions towards accessibility is the best way to ensure your product is (and will continue to be) inclusive. Test sessions not only give you useful feedback about how accessible your product is, but also help build awareness across your team, and encourage them to build more inclusive experiences.

Allow enough time between tests so that you have a chance to take into account any feedback, and also give room for your product to evolve in new ways that may warrant additional testing later on. This also gives room for the attendees at the session to learn more about accessibility within the context of their own work. In our experience, we’ve seen more and more people questioning and reflecting on the impact that their work might have on the accessibility of our products.

Spread the word

We hope that this post was helpful and will encourage you to run some test sessions on your product and maybe even share some knowledge too afterwards. If you’re excited about accessibility and would be interested on joining us into making a better web for everyone, check our careers page!