Using Tableau to prioritize accessibility errors

Tableau is a tool that helps people see and understand their data. As part of a new accessibility program, I utilized Tableau to visually communicate priorities to stakeholders and identify repeated errors, and target quick wins.

Our senior leadership is interested in high level information: progress over time, the general performance of a page, and hot spots.

Developers are interested in fixing errors in code and identifying best practices.

Using Tableau, we can present a high level overview as well as details to keep both parties engaged and informed.

The data

In order to create our Viz, we need a data set. In partnership with another open-source dev, I created a command line tool that runs an audit of the top 100 most visited pages of tableau.com*. To achieve this, we used axe-webdriverjs for the audit, which uses the axe api and selenium webdriver. The results for each page are saved as a JSON file.

Examples:

• tableau.com/about.html corresponds with audit-date/about.json
• tableau.com/about/careers.json corresponds with audit-date/about/careers.json

The directory is structure is important to note when we create our join in Tableau, coming up shortly.

Example Folder Structure of Data:

audits/
├── date/
│   ├── index.json
│   ├── about/
│       ├── careers.json
│       └── contact.json
├── anther_date/
│   ├── index.json
│   ├── about/
│       ├── careers.json
│       └── contact.json

Here’s an example JSON result from one page of our audit. In the JSON example, you’ll see I’m also calculating the total number of errors in each category and saving them to the metadata object in order to access this information easily to create calculations inside Tableau.

Importing data to Tableau

Next, we import the JSON to Tableau. I set my schema to include as few levels as possible to reduce the amount of time Tableau takes to load the data. Since each page has its own JSON data, I create a join by wildcard that includes all sub-folders. I expand the search to the parent folder so all future audits are automatically included as well. See this example in Figure 1.1.

Figure 1.1

Figure 1.2 shows the schema level. Note that the metadata and violations objects are selected. This provides enough information to communicate our main errors but does not include information in passes, incomplete, or inapplicable to prevent overloading the data source.

Figure 1.2

Creating the high level overview

Creating a score:

One of the main asks from our leadership was some sort of “score” to evaluate at a glance how our pages were performing. The idea comes from Google’s Lighthouse analysis tool. The calculation for the score is available online, but for our purposes, we just need a simple calculation that will give us a trend over time. As long as it’s consistent, we can use any proprietary method for our calculation.

I created a calculation by taking the total number of violations for each page and dividing it by the sum of:

• the total Incomplete per page times .15 to weight the value to 15 percent return expected.
• the total Passes per page times .40 to weight the value to 40 percent expected.
• the total Violations per page times .10 to weight violations to no more than 10 percent.

100 - ([Violations per metadata] / (([Incomplete per metadata] * .15) + ([Passes per metadata] * .40) + ([Violations per metadata] * .10)) * 100)

You might have noticed we’re not adding our calculations to a weight of 100. That’s ok. Remember we’re not including the incomplete tests and are only looking to find some representation visually on progress.

Figure 2.1 shows two columns of data: the page path, and the page score. The score is color-coded, to make it easier to identify problem pages at a glance. This Figure is only a partial list. In reality we have this information for all 100 pages run through the audit.

Figure 2.1

Figure 2.2 shows a chart of the calculated score over time. On the x axis is each month we ran an audit: May, June, September, and January. The top headers indicate the year, 2019 and 2020 (for January). The y axis has the score number. Each blue line in the bar indicates one page. Most pages are clustered around 75 to 90 with a few outliers. Our goal is to reach the highest number possible and trend toward most pages reaching 98 or 99 in automated tests.

Figure 2.2

Interpreting the results

The rest of the Viz is a more detailed evaluation of our results. These dashboards help us identify priorities and spot trends in errors.

Overview: Errors Per Page Over Time

To get a high level comparison, let’s take a look at our table of errors and visitors for each url saved for every audit we’ve run.

The table in Figure 3.1 contains the following information:

• The first cell in each row contains the relative Url.
• The second cell in the row contains the number of errors and visitors for May 7, 2019.
• The third cell contains the number of errors and visitors for June 9, 2019.
• The fourth cell contains the number of errors and visitors for September 3, 2019.
• The fifth cell contains the number of errors and visitors for January 6, 2020.

Note that the September 3 error count for the “About” url was 31, but on January 6, the error count was 60.

Figure 3.1

Drill down with HTML of errors by Url

To understand what changed between audits, we navigate to the dashboard “Html of Errors by Url.” In Figure 4.1 the /about url has several new errors. The image shows a tool tip when hovering over the html. The tool tip provides additional information including the impact of the error, the error message, and a link to the page on production. Our page has been recently updated with new headings and images where many of the errors are occurring.

Figure 4.1

Drill down with a graph of errors per page

We can use the dashboard displaying a “Graph of Errors Per Page” in a similar way. Here we can compare the total number of errors on each page and see the changes between specific audits by filtering the date.

Figure 5.1 shows two nearly identical graphs side by side. The graph is a segmented color bar chart sorted by most errors per page to the left and least on the right. We have a legend of error IDs that match each color: yellow for color contrast, gray for heading-order, pink for image-alt and a dozen more. You also see a date filter — the one on the left is set for September 3, 2019 and the one on the right is set for January 6, 2020. The number of errors per Url is slightly more on the left than on the right, indicating fewer errors overall per page.

Figure 5.1

If we filter out the color contrast error the page with the most errors is our updated about page. Selecting the error block on the Viz provides some additional information in the tool tip, as show in Figure x.x. Three errors dominate the page: alt text for images, heading order, and discernible text for links. We can group these errors into action items for our content editors and developers to fix.

The same layout of graph is seen as Figure 5.2 but we no longer have the yellow color-contrast errors loading. The Urls are sorted from most errors to least, left to right. The left most bar is the about Url. By hovering over each of the colors stacked in the bar, the tool tip displays which type of error is associated with that color and Url and the description and number of instances on that page.

Figure 5.2

Finding the lowest hanging fruit

Let’s say we want to target errors as a group instead of by Url. This can be helpful for component code fixes or global elements. For this, we compare the target HTML, error ID, and the number of Urls that contain the code.

The results give us a pattern of code shared among the highest number of pages. For example, in Figure 6.1, the top two errors that occur on every page are landmark errors. Addressing these two pieces of code will affect every page on our site and reduce the total number of errors by two per page.

Figure 6.1

We can continue using this method to find component based errors — errors that occur on multiple pages due to shared code.

In Figure 6.2, we notice the teaser-item component has a heading order error affecting multiple pages. We create a ticket and suggest using our best practices by changing the heading to <h3 class="heading--h5 teaser-item__title> . Once this fix is complete, we re-run the audit and see we have 82% fewer heading-order errors. The remaining errors are located in fields within the Content Management System as hard coded values and can be identified in new tickets for content editors to address.

Figure 6.2

Error Details

For those curious to learn more about each error type, I added dashboards with their WCAG and 508 guidelines.

Figure 7.1 shows three columns of data. The first column contains the ID of the violation such as color-contrast, aria-label, aria-labeleby, and so forth. The second column contains tags associated with that ID such as wcag2aa, wcag143, section508.22.n. And the third column has colorful squares associated with each of the individual pages that contain that error. When the user hovers over the square, a tool tip with information is provided in addition to the id, tag, and url; the description of the error and a message such as “aria-label attribute does not exist or is empty.)

Figure 7.1

Conclusion

These are just a few of the ways I used Tableau to quickly identify and prioritize accessibility tasks for each sprint cycle. I’d love to hear from you if there’s a dashboard you would find helpful in your accessibility workflow. Have you found agile methods that work well with your team?

Working toward an accessible website is an ongoing process. By visibly sharing the results of our efforts, stakeholders gain a more comprehensive understanding of our program.

Footnotes:

*Automated testing only covers roughly 30–40% of the errors on a page, so we consider this a quick first pass. It’s important to communicate to stakeholders the need to include manual testing to get a full picture. More information about manual testing in an upcoming post.

Interact with the Viz: You can see the Viz here on Tableau Public. This link takes you off the current page.