Accessibility Reporting for a Better Web
A Case Study
--
Background
The web has reduced many barriers and enabled many to access information that wasn’t available before. It’s even been called “The Great Equalizer.”
When accessing this information, everyone a range of abilities and has different needs. In 2011, the World Health Organization (WHO) recognized the move towards the social model of disability where “people are viewed as being disabled by society rather than by their bodies.”¹ When creating websites, designers can create barriers — or mismatches — for those with permanent, temporary and situational disabilities. Arguably, everyone is affected at some point in their lives.
When designers intentionally or unintentionally exclude people from society based on a disability, it’s a form of discrimination. This language might sound harsh, but it’s commonplace on the web even if designers have the best intentions.
Accessibility widgets are becoming popular and promise quick fixes to accessibility issues. In practice, these services only pay lip service and don’t fix most underlying issues.
Increasingly site creators are being sued for non-compliant websites. From a compliance perspective, WCAG 2.1 AA is the standard that holds up in court, but compliance is subjective and creating inclusive websites goes beyond compliance. Website creators usually fall into three camps:
- Ignorant of accessibility or don’t care about accessibility
- Want to make websites compliant with WCAG 2.1
- Want to create efficient and effective websites beyond WCAG compliance
Fear of lawsuits and legal pressure can help with at least the first two items. In this project, we worked on an accessibility product that addresses all three. The third item is about creating inclusive experiences and making websites more usable for people with disabilities beyond compliance.
The Problem
In this research, we propose the following question:
How might we make websites more inclusive and accessible by creating a feedback loop between site creators and end users?
Target Audiences
With this project, there are three primary audiences:
- People with disabilities
- Advocates for people with disabilities
- Site creators
Since our problem focused on the feedback loop between people with disabilities and site creators, these are the most important audiences for the research. Through our interviews, we found that advocates are also an important audience and, in some cases, might report issues on behalf of people with disabilities.
Note: we’re not trying to group everyone with disabilities into one group or suggest they have the same needs. In this research, we focused on users who are blind, users who have low vision and users with cognitive disabilities. Further research is needed to address needs of users with other disabilities.
The Solution
We propose a web app that allows users to report accessibility and inclusive design feedback on any other website on the web — a Better Business Bureau for Accessibility.
Users can look up a website and report any accessibility issues they see. The idea is to apply social pressure on site creators to fix their accessibility and inclusive design issues to create a better web for everyone. This will also be a great platform to mobilize accessibility advocates and social pressure could lead to legal pressure.
It might look like we’re shaming site creators into better accessibility, but based on the site creators we’ve talked to, they would welcome a service like this in the spirit of continuous improvement. Site creators working on larger websites with in-house legal teams do worry about the legal ramifications.
Outcome
We ideated on the Better Business Bureau for Accessibility concept and created a web app called AbleList — a play on the term ableist. (Ableism is the discrimination or prejudice against individuals with disabilities.)
A user can report on any accessibility issues they find on any website on the web. Then other users agree or disagree whether it’s an issue and comment on the issue. Site creators can claim their websites, be notified of any accessibility issues on any of their websites and mark any issues as resolved.
Prototype
Research
To become more familiar with accessibility, we used the following for background information:
- Web Content Accessibility Guidelines (WCAG) 2.1 Success Criteria
- Inclusive Design for a Digital World
- Practical Web Inclusion and Accessibility
- De-Puzzling Digital Accessibility webinar by Marsha Schwanke of the Southeast ADA Center
Research Methods
We conducted two subject matter expert interviews with people who work on web accessibility teams and we were able to test our assumptions about larger site creators as well. We interviewed these people:
- Internal Accessibility Team Technical Lead at Google
- Accessibility UX Researcher at Etsy
Then we conducted interviews with two users with disabilities and two users who are advocates:
- Technology teacher who was born blind
- A retiree who has developed macular degeneration
- Music therapist at a school for the deaf and the blind and who has worked with clients with cognitive disabilities in the past
- A house manager for people with developmental disabilities
During the interviews, we tested our assumptions, validated that we’re solving the right problem and tested a few preliminary solutions.
Key Data
During our research, some key facts helped shape the direction of our research:
- 26% (61 million) adults in the US live with a disability (CDC, 2018).
- 97.4% of the top million homepages had detectable WCAG 2 failures and averaged 51.4 detectable accessibility errors (WebAIM Million Report, 2021).
- Testing with users may uncover 45% more accessibility problems than testing with tools (Disability Rights Commission, 2004).
Based on this data, it’s clear that accessible websites are needed and site creators aren’t doing an adequate job of creating accessible and inclusive websites. Even though automated accessibility checkers and widgets are becoming more popular, this data suggests that accessibility checking needs manual checks by real people.
Competitive Analysis
One interviewee thought a solution like ours existed, but we haven’t found any direct competitors.
For indirect competitors, we looked at accessibility widgets and decided not to compete with them. We also looked at feedback forms for reporting accessibility issues on government websites. Some site creators we talked to like the of having an embeddable AbleList widget on their website inviting feedback from users.
For influencers, we looked at other reporting and voting websites, such as the Better Business Bureau, Stack Overflow and Reddit. Based on our user testing, upvoting systems like Stack Overflow and Reddit were too complicated for users. We also looked at community-managed websites like Stack Overflow and Wikipedia. Future research will concentrate on community-moderated posts and spam prevention.
Concept Evaluation
Initially, we planned on creating a widget that would inform users of the accessibility and inclusive design features on a website by showing a scorecard and allowing site creators to collect feedback. We showed interviewees four potential ways to display scorecard information.
After the interviews, it became clear that the scorecard wasn’t a good idea:
- People with disabilities know if the site is accessible fairly quickly without a scorecard.
- Site creators are worried about the legal risk of displaying potential accessibility issues on their site.
- Site creators are also concerned about saying their site is accessible or inclusive when it’s not.
Pivoting Solutions
Every interview participant liked the idea of collecting accessibility feedback in some way. After our initial research, we needed to decide on a direction:
- An internal feedback loop (with or without a scorecard)
- The Better Business Bureau for Accessibility
- Create an accessibility process management app for web teams to use (this came up during one of the interviews)
After our research, we were considering an app to manage incorporating accessibility at various stages of the development process. We liked that this addresses accessibility concerns early and throughout the process.
We decided on the Better Business Bureau for Accessibility idea because everyone we interviewed liked the idea (with some minor concerns) and we liked the social pressure idea to create change.
To create effective change on a wide scale, this was the best solution.
Ideation
After we decided on a direction, we compiled our interview findings and created personas and customer journey maps.
Personas & Customer Journey Maps
We concentrated on individuals with cognitive and visual disabilities based on our interview participants. In the future, we want to expand this to individuals with hearing, motor and emotional disabilities.
Flows & Navigation
The user flows really helped us think about the multiple entry points for users. The sitemap was a very helpful planning tool to visualize the overall scope of the product and wireflows were our first, low-definition glimpse of the key actions on pages.
Early Concepts
These are the first low-fidelity sketches that started to make our nebulous thoughts about features more concrete.
Voting vs. Agree/Disagree
Early on we had fixated on a Reddit-style up-or-down voting system as seen in the wireflows and early concepts.
During the mid-fidelity stage, we were concerned about the usability of the up-or-down voting system, so we created a design with Agree/Disagree buttons instead. After that, we decided to explore other options and decided on the up-or-down voting system (again).
In user testing, the up-or-down voting system was very confusing, so we went back to the Agree/Disagree buttons and tested those, which performed well.
The next image shows the evolution of this feature from early concepts to the final prototype.
Reporting Issues
One of the risks of this project is relying on users to report compliance issues without knowing the compliance rules. While we’re accepting a loose definition of accessibility — if it’s annoying, it’s probably not accessible — we also want to empower users to add more accurate information.
We created a “Similar Issues” section that will intelligently list issues with similar keywords on the same page. This feature will also filter the WCAG success criteria section and only list relevant success criteria.
We also wanted meaningful categories to use as filters on the site. Based on our interviews, users preferred context-based categories (i.e., the context in which someone experienced the accessibility issue). We landed on the categories (seen in the previous image) based on how other accessibility professionals categorize mismatches on websites.
Headings & Region Landmarks
In our research, we discovered that users who are blind use heading navigation most often. More advanced users also use region landmarks to quickly navigate pages as well. We came up with a robust heading and landmark structure that can be referenced during development.
Exploratory Study Testing
Once we had a mid-fidelity prototype, we conducted an exploratory usability study with the following individuals:
- User with low vision
- Standard web user
- Two web developers
Overall, the tests went very well and based on user feedback, we adjusted quite a few text labels and reworked a few sections.
Voting System
One of the biggest insights we gained during testing was that the up-or-down voting system wasn’t working (as previously mentioned). We realized this wasn’t working after two tests, so for the final two tests, we also presented an earlier prototype with Agree/Disagree buttons. With up-or-down voting, some users thought the arrows would change the order of the issue cards while others weren’t sure whether the vote was for agreement or importance.
Tunnel Vision
Our low vision testing participant used a handheld magnifying glass held up to the screen. This along with their disability created tunnel vision and this caused us to rethink the page flow on key pages. Sighted users can see the whole page and focus on key sections with their eyes, but certain disabilities or assistive technologies can result in the opposite: focusing on a specific piece of content without knowing how it relates to the whole. We recreated the page with a stronger page flow and hierarchy.
The page screenshot in the wireframe confused multiple users and they thought this was actual page content, so we put the screenshot in a computer in the final prototype (see previous image).
Visual Design
Branding
Before starting wireframes, we wanted to give our product an identity, so we started thinking about a name. We landed on AbleList, because the sites listed on AbleList are potentially discriminating against people with disabilities. During user testing, participants understood the name was associated with accessibility, but there was some confusion around the name.
We chose Museo Sans Rounded as our typeface, because it’s a very legible font for both headings and body text. It’s human and light-hearted.
We chose blue as our primary color because it has been associated with accessibility for a long time and communicates stability and harmony. Accent colors were chosen with the thought the design would be more colorful, but blue ended up being the predominant color.
Foreground and background colors are WCAG 2.1 AA compliant. Some of the visual elements on this website are very subtle. We need to test that some of the visual elements, such as borders between sections, are WCAG 2.1 AA compliant. In the spirit of being as inclusive as possible, we’re also considering whether we should redesign elements so that they are WCAG 2.1 AAA compliant.
Design System
Starting with the mid-fidelity wireframes, we planned out design atoms using the atomic design methodology.
Design
We decided on a more light-hearted tone for the design to (hopefully) make it fun for users to use. We selected this tone so that it didn’t feel like we were shaming people. Instead, we want people to feel the spirit of continuous improvement.
We designed a Chrome extension so that users can view accessibility issues as they browse the web and make reporting issues easier. In addition to the Chrome extension, we also plan on extensions for JAWS and NVDA, which are the most popular screen readers.
Outcome
Challenges
The biggest challenge was synthesizing the research and deciding on a direction for the solution. As we learned more about accessibility from our interviews and researching the topic, we realized our first idea wasn’t viable.
After pivoting, the Better Business Bureau for Accessibility idea was very fragile at that moment and there were plenty of reasons not to pursue this idea. It’s difficult to evaluate ideas in this fragile state and trust that there is potential.
After careful consideration, the benefits outweighed the risks.
Reflection
The biggest takeaway from this project was that the industry is a long way from truly making websites accessible and even further from making them inclusive and equitable.
We’ve been interested in the idea of designing for the extremes. Seeing our work through the eyes of someone with macular degeneration using a magnifying glass really changed the way we think about designing websites. The idea of seeing a page through tunnel vision and being able to make sense of where you are is a powerful idea.
Sighted users can see the whole and focus on specific sections while users with certain disabilities might start within a specific section and gradually the whole is revealed to them. Meanwhile people with cognitive disabilities need simple layouts with clear steps and large calls-to-action. Based on this, creating a strong visual hierarchy and page flow is important for creating accessible pages.
Finally, this was the second project that we’ve done wireflows and it’s been a very effective tool. It’s freeing to map the whole site structure while seeing the the key actions on each page.
Next Steps
In this research, we didn’t have time to do some of the items we wanted to. These are the items we would like to address next:
- Perform additional interviews with people with hearing, motor and emotional disabilities.
- We tested mid-fidelity wireframes and we would like to test the final prototype — especially the new issue cards and the interactivity with reporting a new issue.
- We want to research community-based moderating models so that the app is scalable without too much overhead.
- During user testing, a participant mentioned having a widget available on their websites to invite feedback on AbleList. We would like to validate this idea with other site creators.
Medium was chosen for this case study so that a wide variety of people regardless of ability or disability would be able to read this content.
Footnotes
[1] World Health Organization & World Bank. (2011). World report on disability 2011. World Health Organization. https://apps.who.int/iris/handle/10665/44575
