<![CDATA[Stories by Craig Morten on Medium]]> https://medium.com/@craigmorten?source=rss-c6b59d857cb7------2 https://cdn-images-1.medium.com/fit/c/150/150/1*A81L1TJFIQZWUMwisqSdGw.png Stories by Craig Morten on Medium https://medium.com/@craigmorten?source=rss-c6b59d857cb7------2 Medium Sat, 16 May 2026 18:07:10 GMT <![CDATA[Automating A11y Testing: Part 3 — Testing Library and Beyond]]> https://medium.com/john-lewis-software-engineering/automating-a11y-testing-part-3-testing-library-and-beyond-f4cd590c7395?source=rss-c6b59d857cb7------2 https://medium.com/p/f4cd590c7395 Wed, 14 Feb 2024 13:58:26 GMT 2024-02-16T17:53:24.533Z Automating A11y Testing: Part 3 — Testing Library and Beyond

Last year I gave a talk on the topic of automating web accessibility (a11y) testing at the John Lewis Partnership Tech Profession Conference 2023 which I have converted into this “Automating a11y testing” article series.

Part 1 of the series covered a number of tools from the Axe suite for static analysis of sites, and Part 2 started to explore some of the non-Axe tooling in the wider a11y test automation space.

In this Part 3 we will start to explore how we can weave accessibility concerns into our day to day unit and integration test automation.

Now remember from Part 1…

It is also key to remember the importance of manual validation and exploratory testing.
At the end of the day it is real people that are visiting and using your site, so it is incredibly important to ensure the quality of their experience, not just to tickbox that automation has passed.

Keeping this in mind, let’s take a dive into something a lot of folks in the front-end world have probably encountered — Testing Library!

Introducing Testing Library

Testing Library is a collection of packages for supplementing test frameworks that allow you to query and assert on your application similar to how your users would use it.

“The more your tests resemble the way your software is used, the more confidence they can give you.”

Instead of testing against abstract components that you write for your framework of choice (like Enzyme and other libraries might do), Testing Library ensures you are testing against the final rendered result of DOM nodes to bring your tests closer to reality.

So what does this have to do with accessibility automation?

Laptop on a desk with code open in an editor.
Photo by James Harrison on Unsplash

The key is in the three groups of utility queries that Testing Library provides:

Queries accessible to everyone

These queries not only reflect the experience of visual users but also those who use assistive technologies to visit your pages:

  1. ByRole
  2. ByLabelText
  3. ByPlaceholderText
  4. ByText
  5. ByDisplayValue

Where you mess up, these queries make it immediately clear that something is amiss, whereas other queries or frameworks may miss mistakes (more on this later!).

Not able to find that button or a crucial piece of copy? With these queries you can be confident when something isn’t right with your code, and that it will be caught with a helpful error message explaining why it couldn’t find what you were looking for and suggestions as to alternative options.

Consider this example React Testing Library test where we have accidentally used the incorrect semantic element for a button:

import React from 'react';
import { render, screen } from '@testing-library/react';

describe('Submit button', () => {
  test('should render a "Submit" button', () => {
    render(<div className="button">Submit</div>);

    expect(screen.getByRole('button', { name: 'Submit' })).toBeInTheDocument();
  });
});

This will result in the following output making it clear that something is inaccessible about our button, in this case because we’ve used a <div> instead of a <button>:

TestingLibraryElementError: Unable to find an accessible element with the role "button" and name "Submit"

There are no accessible roles. But there might be some inaccessible roles. If you wish to access them, then set the `hidden` option to `true`. Learn more about this here: https://testing-library.com/docs/dom-testing-library/api-queries#byrole

Ignored nodes: comments, <script />, <style />
<body>
  <div>
    <div
       class="button"
    >
      Submit
    </div>
  </div>
</body>

How about this example where we have accidentally hidden the button?

import React from 'react';
import { render, screen } from '@testing-library/react';

describe('Submit button', () => {
  test('should render a "Submit" button', () => {
    render(
      <button type="button" style={{ display: 'none' }}>
        Submit
      </button>,
    );

    expect(screen.getByRole('button', { name: 'Submit' })).toBeInTheDocument();
  });
});

Again, unlike other libraries that would probably miss this, Testing Library gives you helpful output to make it clear that the button is still not accessible:

TestingLibraryElementError: Unable to find an accessible element with the role "button" and name "Submit"

There are no accessible roles. But there might be some inaccessible roles. If you wish to access them, then set the `hidden` option to `true`. Learn more about this here: https://testing-library.com/docs/dom-testing-library/api-queries#byrole

Ignored nodes: comments, <script />, <style />
<body>
  <div>
    <button
      style="display: none;"
      type="button"
    >
      Submit
    </button>
  </div>
</body>

By placing the emphasis on querying elements using their accessible role and labels you will naturally start to write components for which you can be confident of the correct element choice and that content will be announced as you expect for all users, no matter how they use your application.

Semantic queries

These queries allow you assert on elements that use HTML5 and ARIA compliant attributes to provide additional context to users:

  1. ByAltText
  2. ByTitle

The use-cases for these are limited — often you can use one of the previous queries to acheive the same thing, for example passing the alt as the accessible name option to the ByRole query. Where your use-case demands it these are still preferable to the next class of query, keeping a focus on the end user and the content they will be presented with.

Test ID queries

Lastly, there is the option to use ByTestId to query elements with data-testid attributes.

These attributes cannot be seen or heard by users in any way so should only be used as a last resort for test cases where matching on the role or text for an element doesn’t make sense.

If you use these queries for testing components that have important value to your users it can be very easy to miss that you’re not using the right semantic element or the text isn’t accessible, giving you false confidence in your code.

For example, let’s consider the submit button again, but using ByTestId instead of ByRole:

import React from 'react';
import { render, screen } from '@testing-library/react';

describe('Submit button', () => {
  test('should render a "Submit" button', () => {
    render(
      <div data-testid="button" style={{ display: 'none' }}>
        Soobmit
      </div>,
    );

    expect(screen.getByTestId('button')).toBeInTheDocument();
  });
});

There are a plethera of mistakes here from incorrect semantics, typos in the copy, and the fact the button isn’t even displayed to any users… but yet this test will still pass!

For this reason you should always prefer the previous accessible or sematic queries.

You can learn more about these queries in the Testing Library docs.

Weaving accessibility into your unit tests

Let’s look at some examples of how we can best use these queries to blend accessibility ideas into our unit tests using the @testing-library/dom package.

~ listed on a website with a No.1 Morello Cherry & Almond Tart added to the trolley.

Example 1: Add to trolley button

Let’s say we had the following button for adding an item to the user’s trolley:

<button type="button" id="add-to-trolley">Add To Trolley</button>

If we were writing a test to assert that it performs the expected action we might do something like:

const button = document.querySelector('#add-to-trolley');

// ... test code to trigger a click on the button and assert
// on the item now being in the trolley.

This will certainly get ahold of the button and allow us to write a test, but it falls down in a number of places:

  1. The id of the button is likely an implementation detail, and certainly not something that users care about. If it were to change the test would fail so we’ve introduced a fragile test!
  2. If we were to swap the <button> for an <a> the test would still pass. Although the behaviour might still kinda work for visual mouse users, you will baffle your screen reader users who will hear that this is a link, and your Safari keyboard users who won’t be able to Tab to the element.
  3. This test fails to take the main user facing information into account — the button text for “Add To Trolley”. It wouldn’t be a happy place if the behaviour you were testing was accidentally on the wrong button because there was a mistake with ids, or if it worked but you had a typo on the button for “Add To Troll” that was missed!
  4. This test would also pass even if the button was hidden using CSS styles 🫣.

With Testing Library we can cover off all these points with a single line swap out:

const button = screen.getByRole('button', { name: 'Add To Trolley' });

// ... test code to trigger a click on the button and assert
// on the item now being in the trolley.

Now not only can we assert on the desired click behaviour but we can also be certain that:

The element is a visible button with an accessible name of “Add To Trolley”.

Example 2: Lazy loaded image

Let’s say we had the following lazy loaded image used on a blog post article tile:

<img
  data-testid="article-1"
  alt="Young West Highland terrier sitting on the floor by a window"
  loading="lazy"
  src="https://unsplash.com/photos/white-long-coated-dog-sitting-on-floor-f5KQq4Wfxg8" />

If we were writing a test to assert that it had expected behaviours such lazy loading, error handling, impression tracking, etc. we might do something like:

const img = document.querySelector('[data-testid="article-1"]');

// ... test code to assert on attributes and behaviours

Again we have successfully retrieved the image element, but we could do better. What if the image is no longer for the first article? What if someone accidentally removed the alternative text in a refactor?

Again we can use Testing Library to decouple us from implementation detail while gaining confidence that the image being asserted on is the real deal:

const img = screen.getByAltText('Young West Highland terrier sitting on the floor by a window');

// ... test code to assert on attributes and behaviours
Note: In this example we’ve opted for ByAltText for demonstrative purposes.
You could also use ByRole with an img role and accessible name to disambiguate between it being an image vs an input, area, or any other elements that can have alternative text.

Not just for unit tests

So is all this goodness reserved just for our unit tests?

Luckily not!

Testing Library boasts a huge number of packages that cover a whole host of different test frameworks from working just with the DOM (as we’ve demonstrated above) to integrating with integration and end-to-end test frameworks such as Cypress:

The eagle-eyed among you may notice that Playwright is missing from the above list.

Don’t panic — in both the Playwright Locators and the new experimental Playwright Component Testing the Playwright APIs include an equivalent set of queries that mirror the Testing Library queries, making it very easy to write in the same style, or even migrate unit test code to end-to-end test code.

Not just queries

Thus far we’ve been focusing a lot on the Testing Library queries and how we can use them to ensure the accessibility of elements in our test automation, but the Testing Library ecosystem also boasts a host of other packages and extensions that can also be used in accessibility testing.

A special mention goes to the @testing-library/user-event package which attempts to provide utilities that simulate the real events that would happen in the browser when users interact.

For example, consider the following which uses the package to type a message into a text input:

const user = userEvent.setup();

await user.type(textInput, 'A message');

// ... remainder of test code

Whereas a lot of testing libraries (or something you wrote yourself) would likely dispatch a single DOM input or change event to the text input, the @testing-library/user-event package goes further and tries to simulate the entire interaction:

  1. A hover event is dispatched to the textInput to simulate moving the cursor over the element.
  2. A click event is dispatched to the textInput to simulate clicking the element, just as a real user would before they start typing.
  3. Characters are then dispatched to the textInput one by one, just as a real user would type, with associated events.

This is far more realistic, and more likely to give you a clear idea how your elements will behave for real mouse, keyboard, and assistive technology users.

Where assumptions are incorrect (e.g. you don’t “hover” with a keyboard or screen reader) the methods are all highly configurable, with a host of lower level APIs for constructing your own interaction flows.

With other useful “convenience APIs” such as user.tab() you can also easily cover off keyboard scenarios for your components the same as you would for mouse based test journeys.

You can learn more about other packages in the Testing Library ecosystem docs.

Testing Library gotchas

As with any tool, there are some potential gotchas with Testing Library to be aware of.

Too much choice

As seen in the second example above, there are often a few ways in which you can use different queries to select the same element.

And with choice comes footguns.

Man staring at laptop deep in thought.
Photo by Wes Hicks on Unsplash

When you are writing your queries take care to be as explicit as possible for your given scenario, while of course being pragmatic. For example, in the first example it might have been tempting to just select on the button role:

expect(screen.getByRole('button')).toBeInTheDocument();

Although we can be confident that we will still be selecting a button element, unfortunately we now have no confidence that the button has the correct accessible name (if any at all) for customers!

Similarly you can be caught out with the likes of ByText:

expect(screen.getByText('Add To Trolley')).toBeInTheDocument();

With this test assertion we can certainly be confident the expected text is on the page, but there is no guarantee that this is actually the button element so there is no protection against future refactors to the code which could see the <button> element replaced by something incorrect such as an <a> element.

As a rule of thumb I would recommend:

  1. Should the element have any semantic meaning that is important to users? Use ByRole with both the role and accessible name.
  2. If not, should the element have text that is conveyed to users? Use ByText or a similar text query with the accessible name.
  3. Otherwise… you should probably challenge why you have the element!

If an element has no semantic meaning and no text content nor accessible label then it’s likely that either:

  • You should be using a different element.
  • You need to give your element text content or an accessible label such as alt , aria-labelledby, or aria-label.
  • You shouldn’t have the element at all!

Testing Library is not a real browser

Although the library does a great job of helping to write accessible components with a user-first viewpoint of writing tests, it does have a shortcoming in that it isn’t an actual browser.

Assistive technologies such as screen readers use a combination of the DOM and accessibility APIs in order to understand an element and convey information to users.

You can learn more about how browsers map HTML elements and attributes to platform accessibility APIs in the W3C HTML-AAM specification.

But much like browsers have varying accessibility compliance and support, Testing Library doesn’t always reflect the role or accessible name of an element accurately. In fact, because Testing Library only has access to the DOM and can’t use these accessibility APIs it has to rely heavily upon some awesome packages such as aria-query and dom-accessibility-api to reflect the WAI-ARIA and ACCNAME W3C specifications, meaning it can easily fall out of sync with the latest standards.

For example, at the time of writing (Feb 2024) the following roles are missing or incorrect in the Testing Library latest stable versions:

Luckily for these roles we should see support coming soon in v10 of @testing-library/dom as I added the missing roles in the latter part of 2023!

As always with open source packages — if you spot something that’s not right raise an issue with helpful comments and steps to reproduce (an perhaps even contribute a pull request with the changes if confident!).

Testing Library also has an active Discord server for asking queries.

Lack of “holistic” user experience insight

Although Testing Library allows you to weave accessibility concerns and testing into your querying of specific elements, as well as covers off some interaction scenarios, somewhere I feel it falls down is “bigger picture” validation of component or whole site accessibility.

Consider the following React example for a product tile on an ecommerce site:

<ProductTile>
  <ProductTitle />
  <ProductImage />
  <ProductPrice />
  <ProductAttributes />
  <AddToTrolley />
  <AddToFavourites />
</ProductTile>

For each individual component you can good confidence that your experience is accessible using Testing Library. However, something I’ve observed with this pattern is that the combination of:

  • Separation of concerns through components; and
  • Only introducing accessibility testing through selecting upon specific elements in isolation to each other;

can lead to a naff user experience where the product name gets announced far too many times:

Jaffa Cakes, heading level 2
Jaffa Cakes, link
Jaffa Cakes, image
Current Price of Jaffa Cakes £1.50, Previous Price of Jaffa Cakes £2.00
Jaffa Cakes are vegetarian
Jaffa Cakes are a best buy
Add Jaffa Cakes to trolley, button
Add Jaffa Cakes to favourites, button

Ok “Jaffa Cakes” is reasonably concise here so you might be forgiven, but if the name of your product or article was longer this would potentially be quite frustrating to navigate through for your users.

It is worth calling out that users of assistive technologies such as screen readers don’t just tab through your content bit by bit— they can navigate by headings, links, controls etc. so some repetition may be required to ensure users who jump straight to “Add to trolley” button have all the context they need – check out the WCAG success criteria 2.4.6 and 2.4.9 for some further reading.

Where appropriate, you can likely give your users the benefit of the doubt that they can remember the product they have navigated to without reminding them every step of the way.

Of course mileage may vary, so always validate with your own users — your use-case may be different!

So as it stands, Testing Library unfortunately isn’t quite the silver bullet for building these holistic pictures in test automation — for that you need to look elsewhere or to manual testing.

If you’re interested in how to make tile / card patterns accessible for things like listing products or articles, check out this awesome Inclusive Components article on Cards.

Beyond Testing Library

Given some of the shortcomings of Testing Library let’s briefly look at some options for augmenting our test automation.

Two people sat at laptops collaborating on a piece of paper between them with lots of diagrams.
Photo by Scott Graham on Unsplash

Getting our hands on the accessibility tree

One of the gotchas mentioned above was that Testing Library is having to simulate the accessible role, name, and value calculation because it can’t use browsers’ accessibility APIs to get the actual result.

One option here is to use the Chrome DevTools Protocol (CDP) to query aspects of the accessibility tree. For example in Cypress you could add a command like:

Cypress.Commands.add("getAccessibilityTree", () => {
  return Cypress.automation("remote:debugger:protocol", {
    command: "Accessibility.enable",
  }).then(() => {
    return Cypress.automation("remote:debugger:protocol", {
      command: "Accessibility.getFullAXTree",
    });
  });
});
Credit to Paul Grenier for this approach. Check out this “Automated UI Testing Methodology You Need To Try” article to explore the idea further.

This would allow you to query or simply snapshot the accessibility tree for your page as a means of testing and validating the actual computed accessible roles and names.

It also gives you some insight into how your component is seen “as a whole” from the perspective of assistive technology users which may help our Jaffa Cakes verbosity problem from a previous example.

You can find out more about the Accessibility options on the Chrome DevTools Protocol docs.

Looking to other testing frameworks, the likes of Puppeteer and Playwright both have a built-in accessibility class that allows you to get a snapshot of the accessibility tree in a similar fashion to our hand-rolled Cypress command above:

const snapshot = await page.accessibility.snapshot();

console.log(snapshot);

Playwright has deprecated this API in favour of promoting Axe for accessibility testing instead — depending on your use-case, this may well make more sense for you.

You can find out more about the Accessibility class on the Puppeteer docs and the Playwright docs.

Using the Accessibility Object Model

The Accessibility Object Model (AOM) is an emerging JavaScript API that will allow developers to access and modify the accessibility tree for a page.

This effort aims to create a JavaScript API to allow developers to modify (and eventually explore) the accessibility tree for an HTML page.

It is currently in early stages with an unofficial draft specification and limitation implementation or usage outside of things like Web Platform Tests.

Nevertheless, this is exciting territory — given this will allow us to access the computed accessible role and name for an element, I can see a future where these APIs get incorporated into the likes of Testing Library!

If you want to make use of these APIs today in your own projects there are some experimental flags that can get you some of the way. In Chromium there is an --enable-blink-features="AccessibilityObjectModel" flag which enables an experimental implementation — tread carefully here as it could be removed any time, the equivalent WebKit experimental flag appears to have already been removed.

Alternatively there is also an --enable-experimental-web-platform-features flag for Chromium browsers that exposes the computedRole and computedName properties on elements which Paul Grenier covers in this “The Automated UI Testing Methodology You Need To Try (Pt. 2)” article.

At the moment the safest path to follow is likely the approach used in the Web Platform Tests which make use of WebDriver commands (think Selenium) in their test driver, as Chrome, WebKit, and Gecko drivers all have support as of early 2023.

Some care needed here: If you explore around you might come across the likes of getComputedRole() method for WebDriverIO. Unfortunately despite looking like it is using these new APIs, at the time of writing it is actually using the same aria-query package as Testing Library and so suffers from the same inaccuracies. Take care not to muddle the WebDriver protocol with the WebDriverIO test framework!

Given this is all very new and experimental, unless you are a test framework maintainer I would recommend relying on the likes of Testing Library for now with the understanding that there may be some role or accessible name inaccuracies — manual testing FTW!

A missing puzzle piece?

With all the tools we’ve covered in these first three parts we’re starting to be in a really awesome position for building out test automation that can really give us confidence on the accessibility of our applications.

However, something the tools we’ve explored still can’t help us with is being able to automate tests for asserting on the quality of user experience for those who are using your application with something other than mouse or keyboard.

Using Testing Library or other packages such as cypress-real-events can allow us to build reasonably realistic tests that can use mouse or keyboard similar to how real users might, but this still leaves a gap for assistive technologies such as screen readers.

Imagine you are building out components that require the coordination of multiple elements such as combobox or disclosure patterns, then Testing Library and other tools can only get you so far:

  • Static analysis can catch any common gotchas and incorrect syntax.
  • Testing Library queries or Accessibility Tree snapshots can give you confidence on usage of the correct accessible roles and names.
  • @testing-library/user-event and similar packages can give you confidence for mouse and keyboard.

But nothing other than manual testing can really provide confidence that the combination of a browser + screen reader + your application is being announced and behaving in the expected and intuitive way for a screen reader…

Or so it used to be the case!

In the next part of this series we’re going to start exploring the cutting-edge field of screen reader standardisation and automation, looking into:

Till then folks — see you soon! 👋

Hi, my name is Craig Morten. I am a senior product engineer at the John Lewis Partnership. When I’m not hunched over my laptop I can be found drinking excessive amounts of tea or running around in circles at my local athletics track.

The John Lewis Partnership are hiring across a range of roles. If you love web development and are excited by accessibility, performance, SEO, and all the other awesome tech in web, we would love to hear from you! Find our open positions here.

Automating A11y Testing: Part 3 — Testing Library and Beyond was originally published in John Lewis Partnership Software Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

]]> <![CDATA[How to debug React hydration errors]]> https://medium.com/@craigmorten/how-to-debug-react-hydration-errors-5627f67a6548?source=rss-c6b59d857cb7------2 https://medium.com/p/5627f67a6548 Sat, 27 Jan 2024 13:55:21 GMT 2024-02-04T19:16:32.639Z http://creativecommons.org/licenses/by/4.0/ Escaping React Hydration Error Hell

Introduction

When hydrating your React applications you might have come across errors that look something like this:

DevTools console error message. Warning: Expected server HTML to contain a matching <button> in <div>.

This error occurs if there is a difference between the React tree that was rendered server-side versus the React tree that is created client-side during the first render in the browser, as React is unable to reconcile the two differing trees in order to take control over the tree’s UI and interactivity.

Here we will look into some common causes, how to debug your error, and some general solutions to avoid hydration errors.

Causes

There are several common causes for hydration errors to be aware of.

Photo by Elisa Ventur on Unsplash

Browser-detection based rendering

If you have any logic that feature detects if a component is rendering in the browser or on the server, then great care is needed. For example:

const isClient = typeof window !== "undefined";

Let’s consider the following contrived example:

const IsomorphicContainer = () => {
  const isClient = typeof window !== "undefined";

  return isClient ? <ClientOnlyComponent /> : null;
};

Where we imagine the <ClientOnlyComponent /> can only be used client-side. This is not uncommon when integrating with third parties scripts — perhaps it’s a client-side GTM DataLayer configuration component.

Issues arise with this code due to React’s understanding of the world being out of sync:

  1. On the server the isClient condition evaluates to false so a null value is rendered.
  2. Client-side prior to hydration, window becomes defined meaning this expression would evaluate to true.
  3. Upon hydration the functional component is rendered for the first time resulting in the <ClientOnlyComponent /> which mismatches with the server-side rendered null value.
  4. This results in a hydration error being thrown.

Breakpoint-detection based rendering

As an extension to the browser-detection issues, any component that attempts to feature detect and render conditionally as a result of browser specific APIs can suffer. For example:

const mediaQueryList = window?.matchMedia("(max-width: 768px)") ?? {};

Let’s consider the following contrived example:

import { useMediaQuery } from 'react-responsive';

const BreakpointVaryingCTA = ({ showModel }) => {
  const isMobileOrTablet = useMediaQuery({ maxWidth: 768 });

  return isMobileOrTablet ? <a href="/content">Additional information</a> : <button type="button" onClick={showModel}>Additional information</button>
};

Here we have a contrived setup where on mobile and tablet devices we are rendering a link, and on desktop devices we are wanting to open a modal for additional information.

Server-side the media-query hook will evaluate to false because the width server-side is not less than 768px — well in fact there simply isn’t a concept of viewport server-side at all!

Client-side on desktop we’re lucky because the hook will also resolve to false and we get a hydration match.

Client-side on mobile we will see issues though. The hook will resolve to true and the first render will result in an anchor element which doesn’t match the button rendered on the server, hence resulting in an error.

Whitespace

A frustrating and nit-pick cause of hydration errors is whitespace mismatches between the server-side rendered content and the client-side rendered content.

Consider the following two variations on a React root element created by string interpolation with some template literals:

<div id="root">${html}</div>
 
<div id="root">
  ${html}
</div>

The latter example will likely result in a hydration error due to a mismatch between the newlines wrapping the React tree content and the root node:

Warning: Did not expect server HTML to contain the text node " " in <div>

Similarly if you perform any minification of the response HTML by stripping whitespace you might find that you get similar issues.

Data differences

If data that is reflected to the user via a component can vary between the server render and client render, either due to timings or the environment, this will also cause a mismatch.

The most common form of this issue is rendering timestamps:

  1. Not only will the client-side timestamp not match the server-side one as some time has passed;
  2. If you server is in a different timezone than the client then without care to ensure timezones are considered in the outputted value you can get differences.

Other examples can be the likes of:

  • Reflecting API data within components that is retrieved prior to hydratation — for example, say you refresh your API data on the client and then hydrate, if the API data has changed since the server-side render then you will get a mismatch.
  • Using non-deterministic ids such as using the uuid package on the server and client — instead look to use React.useId() if you are using React 18 onwards.
  • Character encoding differences — ensure your server and client match! Usually utf-8 is a good bet.

Invalid HTML

Some elements cannot be nesting within other elements. For example you cannot nest an <a> element within another <a> element.

DevTools console error message. Warning: validateDOMNesting(…): <a> cannot appear as a descendant of <a>.

Depending on the browser such invalid elements may be ejected from the DOM prior to hydration resulting in a mismatch when React tries to hydrate.

The key here is to write valid HTML!

Third party interference

There are a couple of known scenarios where mechanics outside of your code can interfere with the server response resulting in hydration issues:

  1. Browser extensions manipulating the page — see this React issue.
  2. Cloud providers / CDNs manipulating the HTML response — see the Cloudflare docs.
  3. Google Chrome translate function manipulating the page — see this React issue.
  4. iOS format detection manipulating the HTML response — see the NextJS docs.
  5. Early running third party scripts, e.g. GTM, HotJar, manipulating the page prior to hydration.

Debugging

So let’s look at a few ways in which we can debug the root cause of hydration warnings!

Photo by Scott Graham on Unsplash

React development build

When using React’s development build you should receive full, non-minified hydration warnings in your DevTools console that allow you to quickly triage which component has the issue.

From there you can quickly pattern match your component against some of the known causes of hydration errors and resolve.

If you are unable to currently run a local version of your application with React in development mode that can integrate with Test or Production environment APIs, it may be worth investing time as a team to introduce this capability for an improvement in developer experience.

Log recoverable errors

These options assumes you are using React 18 onwards.

Vanilla React applications

When calling hydrateRoot() for your React tree there is a third argument you can pass for options.

One of these options is onRecoverableError which accepts a callback to invoke whenever React automatically recovers from an error, for example a hydration error.

It is recommended that you consider adding instrumentation to this callback so that you are able to log these issues to your observability platform, e.g. New Relic, Sentry, Datadog, Elastic RUM etc.

Here is an example snippet:

import { hydrateRoot } from 'react-dom/client';
import MyObservabilityPlatform from 'my-observability-platform';
import App from './App'
 
function onRecoverableError(error, errInfo) {
 let context = {};
 
  if (errInfo?.componentStack) {
     // Generating this synthetic error allows monitoring services to apply sourcemaps
     // to unminify the stacktrace and make it readable.
     const errorBoundaryError = new Error(error.message);
     errorBoundaryError.name = `React ErrorBoundary ${errorBoundaryError.name}`;
     errorBoundaryError.stack = errInfo.componentStack;
 
     // REF: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Error/cause
     error.cause = errorBoundaryError;
 
     // You can also just add the plain text as added event context.
     context.componentStack = errInfo.componentStack;
  }
 
  // Replace with your error monitoring service.
  MyObservabilityPlatform.captureException(error, { context })
}
 
const domNode = document.getElementById('root');
 
hydrateRoot(domNode, <App />, { onRecoverableError });
Credit for the code snippet to the developers at Sentry, see this React issue for the discussion.

More information on the hydrateRoot() options are available on the React docs.

NextJS applications

When using NextJS you are not responsible for invoking the hydration of the application, this is done for you by NextJS.

NextJS does not currently expose the ability to pass an onRecoverableError option to hydrateRoot(). Follow https://github.com/vercel/next.js/discussions/36641 for discussion on attempts to get this option exposed.

Nevertheless we can manually patch the option to be able to debug issues both locally and in Production. One way of doing this is using Chrome Local Overrides:

  1. In Chrome, navigate to the page with the hydration errors and open DevTools.
  2. Open the Sources tab and within the left side menu choose the Page tab. Right hand click on the asset tree to reveal a Search in all files option.
  3. In the pane that opens, search for the term onRecoverableError to reveal all code matches.
  4. One of the code hits should be something similar to this.onRecoverableError = a;. We will be patching this line to instead log both the error and the errorInfo arguments.
  5. Right hand click on the file tab and click the Override Content option. This should open up the Overrides tab on the side menu. Ensure the Enable Local Overrides option is checked.
  6. If the file appears minified at this point, use the prettifier button {} to make the code more readable.
  7. Replace the code with the following snippet:
    this.onRecoverableError = (error, errorInfo) => { console.error(error, errorInfo); }. This will ensure that both the hydration error and the additional componentStack information is now logged to the console. Make sure you save!
  8. Refresh the page and observe that you should now have an additional object logged alongside the hydration errors. You can now use the component stack to triage which component has the issue. Because the stack is minified it may be difficult to identify exactly what components are in play — it is often useful to start from an identifiable element such as a main tag and then follow the tree into the children.

An alternative strategy to manually patching is to patch in code, see this NextJS discussion thread or this code example for ideas on how to patch NextJS. This is not advised for Production environments, but might be a convenient capability to introduce for local development builds for improved developer experience.

DevTools debugger

Another way to identify hydration errors can be to use the DevTools debugger to pause on exceptions, particularly “Pause on caught exceptions” can be particularly useful for hydration errors.

The downside to this approach is that it can be quite laborious to identify the cause depending on how many exceptions your application raises (and catches). You might you need to step through quite a number of irrelevant caught exceptions before you encounter something relevant to your issue.

More information on caught exceptions for Chrome DevTools can be found in the Chrome developer docs.

Solutions

Assuming reconciling what is rendered server and client-side is not an option there are few tactics for working around hydration issues.

Photo by Clément Hélardot on Unsplash

Defer to second render

One workaround is to ensure that the same content is rendered server-side and client-side for the first render, and then from the second render onwards we can render the client-side specific components.

const IsomorphicContainer = () => {
  const [isClient, setIsClient] = useState(false);
 
  useEffect(() => {
    setIsClient(true);
 
    return () => {
      setIsClient(false);
    }
  }, []);
 
  return isClient ? <ClientOnlyComponent /> : null;
}

Effects don’t execute server-side so the isClient value is false when rendered on the server.

Effects also run after the first render, so on the first pass client-side the isClient boolean is still false and we have a hydration match.

The mounting of the component to the React tree then triggers the side effect which toggles the boolean to true and yields the <ClientOnlyComponent />.

More information is available on the React docs.

Additional considerations

Performance

This workaround results in the component rendering twice whenever it is mounted to the React tree which has a performance overhead compared to simply rendering once when hydrating.

The additional CPU time spent re-rendering the component can have a negative impact on CPU bound performance metrics such as Largest Contentful Paint (LCP) and Interaction to Next Paint (INP).

Flash of unstyled content

In the case of components where something is rendered visually, if you are encountering a Flash of unstyled content (FOUC) with this approach consider using useLayoutEffect as this will trigger prior to the browser repainting the screen.

This ensures the entire cycle of first render, hook firing, and second render all happens prior to the first repaint to the screen.

This does incur a performance impact on the time to next paint so use of useEffect should always be considered first, especially as metrics such as INP become more prevelant.

NextJS dynamic package “magic”

If you are using NextJS there are some alternative options to specify that a component should be considered client-side only.

This is through making use of the next/dynamic module:

import dynamic from 'next/dynamic';
 
const ClientOnlyComponent = dynamic(() => import('../components/Component'), { ssr: false });

More information is available on the NextJS docs.

Loadable components package “magic”

If you are using the @loadable/component package there is an option to specify that a component should be considered client-side only, similar to the next/dynamic package:

import loadable from '@loadable/component';

// This dynamic import will not be processed server-side
const Other = loadable(() => import('../components/Component'), { ssr: false });

More information is available on the Loadable components docs.

Multiple rendering

When tackling scenarios such as the breakpoint based rendering, you can opt to use the “defer to second render” technique, but this will almost always result in at least one of your breakpoints having a flash of content that is meant for a different viewport before the correct components get rendered on second pass.

This can be mitigated through CSS by visually hiding the content altogether until the second render, but this will likely have a negative impact on your Content Layout Shift (CLS) metric which impacts Search Engine Optimisation (SEO) and also lends itself to a poor user experience.

The generally accepted solution here is to instead render all of the potential variations server-side:

  1. Server-side render all of the variations required across the breakpoints.
  2. Use CSS media queries delivered in a render blocking style or link tag that ensures only the desired variation is displayed to users on the client when first loading.
  3. Use the “defer to second render” technique to ensure there are no hydration mismatches.
  4. On second render you can have the undesired components render null and unmount themselves.

Although this does result in an inflated server rendered HTML that might impact your Time to First Byte (TTFB), and it also suffers from the CPU usage caveats of the “defer to second render” technique, it ensures that there is a seamless transition between server and client without any flashes of undesired content.

More information on this technique can be found in this article: Eliminating CLS when using SSR for viewport specific responsive designs.

React tree pruning

This is an advanced solution that is applicable to breakpoint based rendering, tred carefully!

The idea here is similar to the “multiple rendering” technique, but instead of allowing multiple variations to all be hydrated and then removed on second render, we attempt to prune the React tree before / as we hydrate the tree:

  1. Render all variations server-side.
  2. Use CSS to ensure no flash of undesired content.
  3. On hydrate / first render of variations we determine if the variation is desired, and if not, we prune it from the DOM before React get’s a hold of it — React can’t mismatch on a DOM node that it doesn’t know ever existed.
  4. Because all that remains in the DOM is the only valid variation we don’t need to worry about using the “defer to second render” technique, we can simply render what we want immediately.

Similar to the “multiple rendering” technique this does also result in an inflated server rendered HTML that might impact your TTFB, but it doesn’t suffer from any of the other performance impacts of other techniques.

More information on this technique and how to implement can be found in this gist example and this pull request for the @artsy/fresnel package.

You can also check out recent discussion on X (formally known as Twitter).

Seb ⚛️ ThisWeekInReact.com on Twitter: "🤯 React hydration perf trickPermits to avoid hydrating a mobile component on a desktop (or the opposite)Better than hydrating both and hiding one with CSS 👎How it works? Deletes the useless DOM element before hydration.Is this even ok to do so? https://t.co/q53ZOaodDq pic.twitter.com/aWOpQe8vyA / Twitter"

🤯 React hydration perf trickPermits to avoid hydrating a mobile component on a desktop (or the opposite)Better than hydrating both and hiding one with CSS 👎How it works? Deletes the useless DOM element before hydration.Is this even ok to do so? https://t.co/q53ZOaodDq pic.twitter.com/aWOpQe8vyA

Disable hydration warnings

HTML element JSX accepts a suppressHydrationWarning boolean in React that can be used to supress warnings arising from hydration errors.

This should only be used as a last resort in scenarios where the data cannot be reconciled, for example when writing timestamps or dates to the page, and only when the differences being masked are in text content.

More information is available on the React docs.

That’s it folks! Thanks for reading ☺️

Have any other examples that result in hydration errors? Or have any handy ways to debug or fix issues?

Let me know in the comments!

]]> <![CDATA[Automating a11y testing: Part 2— Beyond Axe]]> https://medium.com/john-lewis-software-engineering/automating-a11y-testing-part-2-beyond-axe-af0f32f366e7?source=rss-c6b59d857cb7------2 https://medium.com/p/af0f32f366e7 Wed, 05 Jul 2023 15:04:54 GMT 2023-07-05T15:04:54.189Z Earlier this year I gave a talk on the topic of automating web accessibility (a11y) testing at the John Lewis Partnership Tech Profession Conference 2023 which I’m delighted to be sharing here in an article series format.

In Part 1 of this series I covered a number of tools from the Axe suite for static analysis of sites to find a11y violations, from framework specific packages to VSCode integrations, and sharing a few learnings with caveats and gotchas from using Axe tools.

However, this is just a small subset of available test automation tools. In this article we will start to explore some of the non-Axe tooling (though Axe will certainly pop up again!) in the wider a11y test automation space. Let’s get going!

Linting for a11y

Although some aspects of web accessibility require manual validation as they are subjective regarding the user experience, many requirements of the Web Content Accessibility Guidelines (WCAG) are for ensuring you use the correct semantic markup and associated attributes to correctly present your content to users. Such requirements are well suited to static analysis checks as they are deterministic and rule based — indeed there is such a set of official rules set out in the Accessibility Conformance Tests (ACT) specification which are used by static analysis tools such as Axe that we covered in Part 1.

“Throughout this series I will be mostly focusing on the automation side of a11y testing, but it is also key to remember the importance of manual validation and exploratory testing” — Key considerations from Part 1 still apply!

Although we can run static analysis checks in CI, as a form of push left and also early feedback, a really natural and effective place to run these checks is in code linting. What is linting if not static analysis of code / markup! We’ve already seen the VSCode Axe Linter in the previous article, let’s look at some others.

Eslint

If you are already plugged into the Eslint ecosystem, there are a number of plugins that you can reach for depending on your framework:

Mileage may vary with coverage from plugin to plugin, but each comes with the same great developer experience of real time feedback as you write your code as well as all the other awesome stuff you get with using Eslint such as IDE integrations with intellisense, auto-fix, etc.

VSCode tab open on a “ProductPod Slidedown” React component. In the component a div element is being rendered with a number of event handler props for mouse enter, click, and key down. The entire div JSX is underlined in yellow and there is a hover tooltip with a warning reading: “Avoid non-native interactive elements. If using native HTML is not possible, add an appropriate role and support for tabbing, mouse, keyboard, and touch inputs to an interactive content element.”
A warning from eslint-plugin-jsx-a11y in VSCode

Same as your other linting rules, you can also lean into the power of running the linting both locally, whether manually or through git hooks, as well as in CI to ensure you catch issues before you even consider raising that pull request.

Svelte

Those reading closely may have noticed that the Eslint plugin list above (which was by no means exhaustive) was missing the popular framework Svelte.

“SvelteKit strives to provide an accessible platform for your app by default. Svelte’s compile-time accessibility checks will also apply to any SvelteKit application you build.” — https://kit.svelte.dev/docs/accessibility

Svelte (and SvelteKit) have a number of accessibility warnings built into the compiler itself which cover a subset of a11y rules. It’s super cool to have these built into the framework itself, but it is worth flagging that these aren’t a catch-all. Even within the limited scope of static analysis checkers, as flagged by one of the Svelte maintainers Geoff Rich, there are a number of gotchas around these compiler checks.

For example, the Svelte compiler is only able to check hard-coded values in markup. If you provide a dynamic value via a variable the Svelte compiler isn’t able to determine the possible values for that attribute and thus doesn’t yeild the desired warning.

<script>
    let href = '#';
</script>

<a href={href}>More Information</a>

In this example the href="#" should normally triggers an a11y warning from the compiler, but using the variable results in the warning being surpressed.

Other

There are a limited number of other linting options out there for a11y, some of which are paid for options:

  • ember-template-lint — A linter for Ember projects that includes a11y rules.
  • AccessLint — A paid for (with free trial) GitHub App Integration: “AccessLint brings automated web accessibility testing into your development workflow.”
  • Axe DevTools Linter — In addition to the VSCode plugin covered in Part 1, Axe also have paid for (with free trial) integrations with GitHub Actions, SonarQube, a REST API, and a CLI for their a11y linter.

Visual testing

Let’s pivot over to some ideas around a11y visual testing!

Emulated vision deficiencies

Back in Chrome 83 the Chromium team released a DevTools change that allowed you to emulate different vision deficiencies in the browser. This is accessible from the Chrome DevTools Command Palette (open DevTools and key CMD+SHIFT+P on Mac or CTRL+SHIFT+P for Windows) by searching for “Rendering”, where you can then pick from six different emulated vision deficiencies.

Chrome open on Waitrose №1 Dine In Meal Deal web page. Chrome DevTools is open on the “Rendering” drawer. Under “Emulate vision deficiencies” the “Blurred vision” option has been selected in the dropdown.

Something that I’ve learned recently is that Chrome also boasts a Chrome DevTools Protocol (CDP) that allows you to set almost every feature you get in DevTools via an API. For example, for those using Playwright this example demonstrates how we can set up a CDP session and request to set an emulated blurred vision:

type EmulatedVisionDeficiency =
  | 'none'
  | 'blurredVision'
  | 'reducedContrast'
  | 'achromatopsia'
  | 'deuteranopia'
  | 'protanopia'
  | 'tritanopia';

test('visually acceptable when have blurred vision', async ({ page }) => {
  const client = await page.context().newCDPSession(page);

  await client.send('Emulation.setEmulatedVisionDeficiency', {
    type: 'blurredVision',
  });

  // ... visual test with Playwright / other third party lib
});

Provided your test framework supports a Chromium based browser then CDP should be available and you can make use of this capability in your based visual tests, e.g. using Playwright screenshots or Cypress image snapshots.

For example, here is emulated achromatopsia (where you can’t perceive colour) for a snapshot test on John Lewis Women’s Dresses product listing page (PLP):

VSCode window with two tabs open. One containing Playwright tests for setting up different vision deficiencies via CDP and then using the Playwright screenshot API. The other tab shows the resulting screenshot for the achromatopsia test — a greyscale image of the John Lewis Women’s Dresses product listing page.

Here’s the same PLP from the blurred vision test:

The John Lewis Women’s Dresses product listing page as viewed by a user with blurred vision.

And finally one from the protanopia (where you can’t perceive Red light) test:

The John Lewis Women’s Dresses product listing page as viewed by a user with protanopia.

I wouldn’t recommend going overboard with running all of these variations for every test — visual tests are typically heavy and slow by nature, and fragile to browser upgrades, anti-aliasing, and other false positive issues (although this is somewhat mitigated through using third party vendors who typically have intelligent diffing algorithms).

Mileage may vary with third part integrations — if your provider performs the snapshot on the browser instance you’re running (locally or in CI) then this may well be an option, but if they do HTML + CSS snapshotting to recreate the page in different browsers on their servers then this is likely not a goer.

It could be worth considering adding to a couple to your suites for golden path smoke tests. Alternatively, it can be nice to have such tests running in production on a nightly, or weekly, where the snapshots form the basis of design review discussion opposed to a blocking gate in release.

“In the UK, more than 2 million people are living with sight loss. Of these, around 340,000 are registered as blind or partially sighted.” — https://www.nhs.uk/conditions/vision-loss/
“Globally, at least 2.2 billion people have a near or distance vision impairment.” — https://www.who.int/news-room/fact-sheets/detail/blindness-and-visual-impairment
“Colour blindness (colour vision deficiency, or CVD) affects approximately 1 in 12 men (8%) and 1 in 200 women in the world. In Britain this means that there are approximately 3 million colour blind people (about 4.5% of the entire population), most of whom are male. Worldwide, there are approximately 300 million people.” — https://www.colourblindawareness.org/

The key is to be aware and empathetic in your design and implementation towards folks who have visual impairments, and understanding of their experience of your site. A few extra, simple tests can help bring colour (pun very much intended) to otherwise dry requirements around font-size and colour contrast which are hard to appreciate in isolation — a quick change to a base font-size in your design system could be a game change for some folks experience on your site.

Automating zoom

Another important aspect of visual accessibility is supporting higher zoom levels, typically of up to 200%. This is laid out in several success criteria in WCAG for text resize and content reflow.

“This Success Criterion helps people with low vision by letting them increase text size in content so that they can read it.” — https://www.w3.org/WAI/WCAG21/Understanding/resize-text.html

Unfortunately most major frameworks don’t support APIs to instrument browser zoom at the moment. Searching around you might see references to the --force-device-scale-factor=2.0 device scale factor flag for Chrome, but this relates to pixel density and is not for zoom (can be passed to launchOptions.args array within your framework’s browser setup options if you want to try and see).

This limitation has been flagged for some testing frameworks so we can hope it might be supported in future. For now you can upvote the likes of this Playwright issue for adding an option for browser zoom.

While we wait for frameworks to catch up, the best solution that I’ve found for the time being is to simulate zoom by adding a before hook in your tests which sets the zoom style of the body. For example, this is a snippet from a Playwright visual test in which the a 200% zoom is applied through CSS:

await page.evaluate('document.body.style.zoom=2.0');
Waitrose The King’s Coronation Jewel The Jack Russell Cake product detail page with 200% zoom on desktop.

In many cases this isn’t necessarily representative of using the native browser zoom feature, nor the OS level zoom features for enlarging text, which is what the WCAG guidance is really for. Nevertheless, it somewhat provides an idea as to how your page behaves when magnified, and can be a useful addition to your golden path test suite to understand if critical functionality still works.

If anything this goes to show how important it is to keep up manual and exploratory testing when it comes to a11y, there is no way to get the coverage otherwise! When it comes to vision deficiencies we are really still quite limited on how much we can shift to automation — some smoke tests on Chrome are great, but what about other browsers? What about tablet and mobile?!

If anyone has any ideas on automation in this space I would love to hear them!

Time to get funky

In this last section I wanted to share one last tool called Funkify.

“Funkify is an extension for Chrome that helps you experience the web and interfaces through the eyes of extreme users with different abilities and disabilities.” — https://www.funkify.org/

It is questionably accessibility test automation, as it is a browser extension… but I’ve decided it qualifies as something that automates customer personas to supplement your exploratory testing. It’s also just very cool so worth the share on that premise alone.

Because it is rather good unfortunately (but quite understandably) the maintainers have introduced a premium level around a lot of the functionality. But even if you just have a dabble with the free features I think it’s well worth it for expanding horizons.

Waitrose groceries homepage open in a Chrome browser. In the toolbar the Funkify extension has been activated, displaying a popup with a number of personna simulators that can be selected: “Blurry Bianca, Color Carl, Dyslexia Dani, Trembling Trevor, Tunnel Toby, Peripheral Pierre”.

The extension works well on most sites, and as you can see, provides a series of simple personas or modes that you can adopt while browsing the site. Everything from vision defects to simulated dyslexia with scrambling letters to trembling hands.

For example, here is the Waitrose groceries landing page with the Dyslexia Dani persona example where we can get a flavour of what the user experience might be like (be wary this is an example simulation, it is not necessarily representative of the experience for all folks with Dyslexia):

Waitrose groceries homepage open in a Chrome browser. In the toolbar the Funkify extension has been activated, displaying a popup where the “Dyslexia Dani” persona has been activated. The letters in the text on the Waitrose page is slightly jumbled.

One option that really struck home for me is the Trembling Trevor persona which simulates how it might be like to use the site if you suffer from motor degenerative disorders such as Parkinson’s. Unfortunately Medium doesn’t support embedding videos directly to show this off, so I really encourage giving it a go with the free trial at least once! Once you’ve had a play with the extension on desktop, have a think about touch devices — ask yourself how confident are you with your site’s buttons, links, and dropdowns being usable on a small touch screen?

Closing notes

In this article I’ve covered off a few additional tools and techniques for expanding a11y coverage, exploring extensions to the likes of visual testing and how this can supplement manual and exploratory testing.

This article hasn’t been extensive by far, but hopefully a flavour of what tooling is out there. If you’re keen to explore further, these are some awesome sites where you can learn more and find other tools in the a11y automation space:

Stay tuned for part 3 of this series where we will take a look at the popular “accessibility first” Testing Library framework and start to explore the emerging field of screen reader automation tooling. See you soon! 👋

Hi, my name is Craig Morten. I am a senior product engineer at the John Lewis Partnership. When I’m not hunched over my laptop I can be found drinking excessive amounts of tea or running around in circles at my local athletics track.

The John Lewis Partnership are hiring across a range of roles. If you love web development and are excited by accessibility, performance, SEO, and all the other awesome tech in web, we would love to hear from you! See our open positions here.

Automating a11y testing: Part 2— Beyond Axe was originally published in John Lewis Partnership Software Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

]]> <![CDATA[Eliminating CLS when using SSR for viewport specific responsive designs]]> https://medium.com/@craigmorten/eliminating-cls-when-using-ssr-for-viewport-specific-responsive-designs-2d5ab0d05bfa?source=rss-c6b59d857cb7------2 https://medium.com/p/2d5ab0d05bfa Fri, 30 Jun 2023 18:23:36 GMT 2024-02-02T12:01:38.388Z How to eliminate layout shifts when doing SSR in React

A common gotcha when building some responsive sites is that the desired designs for different viewports are sometimes quite different. In many cases, utilising CSS media queries is sufficient to be able to cater for these differences, but there are occasions where this isn’t necessarily sufficient on it’s own. It is also not always a pragmatic approach to refuse to implement such designs!

A stack of engineering books, a phone, and a plant. The top book is titled “Stunning CSS”.
Photo by KOBU Agency on Unsplash

CSS only gets you so far

For example, take the following markup for a ficticious article component where on desktop information about the author is contained within a slidedown from the top of the component, and on mobile it is inside a static section at the bottom:

<Article>
  <ArticleAuthorInformationDesktop />
  <ArticleImage />
  <ArticleHeading />
  <ArticlePreviewText />
  <ArticleAuthorInformationMobile />
</Article>

Now if the markup used for the “top” desktop and the “bottom” mobile author information components were exactly the same you could instead opt to make use of CCS grid or the CSS flexbox order to simply have the component once in markup, and use CSS media queries to set the appropriate CSS rule to position the component as desired for the different viewports.

However, this comes with caveats:

“Use of the order property has exactly the same implications for accessibility as changing the direction with flex-direction. Using order changes the order in which items are painted, and the order in which they appear visually. It does not change the sequential navigation order of the items. Therefore if a user is tabbing between the items, they could find themselves jumping around your layout in a very confusing way.” — https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_flexible_box_layout/Ordering_flex_items#the_order_property_and_accessibility

If we were to be clever here and try to achieve the desired differences in layout entirely with CSS, we would be introducing potential accessibility (a11y) issues as the experience for visual and non-visual users would differ.

In theory this can be worked around with either clever use of tabindex and / or aria-owns where in either case you can identify the order in which elements are expected to be traversed — however both cases are almost always strongly advised against. It is risky to try to hijack the default tab order of a page, and aria-owns is not supported by VoiceOver meaning it’s use would only resolve the issue for non fruit related devices. Both options are incredibly fragile to changes to associated components where there is a real maintenance overhead to ensuring this is never regressed.

This also all falls apart if the desktop and mobile versions of the component need to be completely different, e.g. a slidedown vs just a static content section — you can’t use CSS to change HTML markup! This means you will almost certainly be needing different components and then either not rendering or hiding the one that doesn’t fit the viewport.

Server-side rethinkings

This is fine right? Well, it is until you start server-side rendering (SSR) this article component. The server doesn’t have a concept of a viewport out of the box, so there is no way to know which of the two variations will need to be in the HTML response that you’re sending to your users’ browsers.

Laptop with a CSS file opened in a code editor.
Photo by Jantine Doornbos on Unsplash

In the absence of viewport details on the server when rendering, this means:

  1. You can guess (ahem, I mean “adopt mobile-first”), but this will inevitably result in some sort of flash of unstyled content (FOUC) or cumulative layout shift (CLS) whenever you guess wrong, which is a bit naff!
  2. You can choose to not render either server-side, but then you will almost definitely get a CLS when the component is hydrated client-side, or best case it will pop in late which is annoying.
  3. You can choose to render both, but then there’s the same issues as above where one of the variations is there on page load and then get’s removed shortly after either looking visually poor and potentially causing a CLS.

This isn’t necessarily a new nor unsolved problem. Aside from the options above, one idea is to make use of the Client Hints to provide your server with context of where this component will be rendered so an appropriate choice can be made. Alternatively a similar flow can be achieved with “user-agent sniffing” through packages such as ua-parse-js where you detect the user-agent header and from that deduce a likely device type and size.

Be sure to take care that you include the source of the hint in any cache-keys you might have otherwise you might cache poison if you render differently for different viewports!

Lastly, perhaps my preferred solution is (1) to use one of the above hint techniques if possible (and reliable) and then (2) do the following:

  1. If you have a trusted hint then render only what is needed in your HTML and stop here.
  2. Otherwise server-side render both variations of the component in your HTML response.
  3. In addition to this double render, add a small snippet of CSS and appropriate styles / classes to the components with CSS media queries ensuring that client-side they are only displayed if the viewport matches.
  4. Upon hydration you can let JS take over, remove the component from the DOM which isn’t required for the viewport and strip the “SSR smoothing over CSS” attributes as they are no longer required.

If you want to see an example implementation of this idea check out the @artsy/fresnel package (and I’m sure there are others).

How can I do this myself?

It’s not always appropriate to start pulling in more packages to achieve a goal, certainly if you’re keeping performance in mind (with the likes of bundlephobia), and said package may be doing more than you need.

I was in this situation recently — the project was already using react-responsive, a wrapper around the matchMedia API with some server-side options for fallbacks, which is only useful if there is a sensible default (uncommon) or if you are using hints to be able to pass through to the package. This doesn’t do enough as I didn’t have hints to work with and pass to the package, so more work was needed to enable CLS-less SSR.

Here’s the first pass solution I settled on for my use-case (subject to change, and mileage may vary):

// index.tsx

import React, { useEffect, useState } from ‘react’;
import { useMediaQuery } from ‘react-responsive’;
import { BREAKPOINTS, BreakpointWidth } from ‘./breakpoints’;
import styles from ‘./index.css’;

interface SSRMediaQueryProps {
  maxWidth?: BreakpointWidth;
  minWidth?: BreakpointWidth;
}

interface DisplayValueProps extends SSRMediaQueryProps {
  breakpoint: BreakpointWidth;
}

const getDisplayValue = ({ breakpoint, maxWidth, minWidth }: DisplayValueProps) => {
  if (typeof maxWidth === ‘undefined’ && typeof minWidth === ‘undefined’) {
    return undefined;
  } else if (typeof minWidth === ‘undefined’) {
    return breakpoint >= maxWidth! ? ‘none’ : undefined;
  } else if (typeof maxWidth === ‘undefined’) {
    return breakpoint < minWidth ? ‘none’ : undefined;
  }

  return breakpoint < minWidth || breakpoint >= maxWidth ? ‘none’ : undefined;
};

/**
 * This component serves to allow viewport specific components be server-side
 * rendered without causing a CLS.
 */
export const SSRMediaQuery: React.FC<SSRMediaQueryProps> = ({ children, maxWidth, minWidth }) => {
  const nonInclusiveMaxWidth = maxWidth ? maxWidth - 1 : undefined;
  const [isClientSide, setIsClientSide] = useState(false);
  const isMatch = useMediaQuery({ maxWidth: nonInclusiveMaxWidth, minWidth });

  useEffect(() => {
    setIsClientSide(true);

    return () => {
      setIsClientSide(false);
    };
  }, []);

  if (!isClientSide) {
    const style = {
      '--xs': getDisplayValue({ breakpoint: BREAKPOINTS.xs, maxWidth, minWidth }),
      '--sm': getDisplayValue({ breakpoint: BREAKPOINTS.sm, maxWidth, minWidth }),
      '--md': getDisplayValue({ breakpoint: BREAKPOINTS.md, maxWidth, minWidth }),
      '--lg': getDisplayValue({ breakpoint: BREAKPOINTS.lg, maxWidth, minWidth }),
      '--xl': getDisplayValue({ breakpoint: BREAKPOINTS.xl, maxWidth, minWidth }),
    } as React.CSSProperties;

    return (
      <div className={styles.mediaQueryContainer} style={style}>
        {children}
      </div>
    );
  }

  if (isMatch) {
    return <>{children}</>;
  }

  return null;
};

Where the imported CSS file is roughly (I’m using SCSS mixins to avoid writing long-hand!):

/* index.css */

@media (max-width: 543px) {
  .mediaQueryContainer {
    display: var(--xs, unset);
  }
}
@media (min-width: 544px) and (max-width: 767px) {
  .mediaQueryContainer {
    display: var(--sm, unset);
  }
}
@media (min-width: 768px) and (max-width: 991px) {
  .mediaQueryContainer {
    display: var(--md, unset);
  }
}
@media (min-width: 992px) and (max-width: 1199px) {
  .mediaQueryContainer {
    display: var(--lg, unset);
  }
}
@media (min-width: 1200px) {
  .mediaQueryContainer {
    display: var(--xl, unset);
  }
}

And the usage is something like:

<Article>
  <SSRMediaQuery minWidth={BREAKPOINTS.sm}>
    <ArticleAuthorInformationDesktop />
  </SSRMediaQuery>
  <ArticleImage />
  <ArticleHeading />
  <ArticlePreviewText />
  <SSRMediaQuery maxWidth={BREAKPOINTS.sm}>
    <ArticleAuthorInformationMobile />
  </SSRMediaQuery>
</Article>

Walking through the implementation:

  1. We first assume that we are always in a server-side or pre-hydration world until an effect (which only runs client-side) tells us otherwise.
  2. In this server-side world we always render the provided children, and furthermore, do so with them wrapped in a <div> that has appropriate styles associated with it to ensure it is only displayed to users for on the desired viewports. Here I was feeling fancy so made use of CSS variables to determine how the display value is set for the wrapper <div>, defaulting to unset if no value is passed.
  3. Once we hit the client we can be safe in the knowledge that the CSS media queries on the wrapper <div> will kick in meaning that although we’ve taken a slight performance hit shipping both variations of component, we don’t suffer from and visual quirks or CLS.
  4. Post hydration we’ll still be in a state where both variations exist — indeed both variations will render on first pass so care is needed to make sure your components don’t have “onload / onmount” like side effects which could “double fire”. (For further reading on why we do this double pass see this GitHub issue. There is a better alternative which is more involved that I’m not covering here – this uses suspense boundaries and manual pruning of the DOM to remove the unwanted tree prior to hydration)
  5. Following the first render the effect will fire, this will update the state to switch to the client-side mode. This triggers a re-render where we either continue to render the children as there is a viewport match, or nothing otherwise.

And just like that we’re free of CLS and can gracefully handle this kind of markup difference between different viewports 🎉

Parting thoughts

The unfortunate state is that there is no perfect answer here, just trade-offs:

  • Get exactly what you want, but having to trust client hints which might not always be there… so maybe only sometimes getting what you want?
  • Have faster time to first byte (TTFB) and other similar page load metrics at the cost of having an annoying CLS vs almost certainly having CLS at the cost of page weight bloat slower that first impression.
  • Potential search engine optimisation (SEO) gotchas as a result of double rendering content (although it appears generally accepted that you shouldn’t get penalised for this kind of behaviour) coupled with the performance issues above for core web vitals having a direct impact on SEO.
  • Care still need for side-effects in SSR double rendered components (unless you put in more work).

Always evaluate what works for your use-case!

  • If you’re in an SPA world where you only client-render, this article was not for you!
  • If you are in a SSR world but the component that can be rendered differently is only rendered upon customer interaction or some other lazy or deferred trigger, then you don’t need to worry — just use client-side techniques.
  • If you’re in a SSR world, maybe consider some of the techniques discussed.

Hopefully this can save some research and thoughts for others in future. If you have any ideas, suggestions, or great ways (or packages) you use to solve this problem I’d love to hear it in the comments!

Like what I have to say? Consider a follow here or on Twitter.

]]> <![CDATA[Automating a11y testing: Part 1 — Axe]]> https://medium.com/john-lewis-software-engineering/automating-a11y-testing-part-1-axe-ed3d215de126?source=rss-c6b59d857cb7------2 https://medium.com/p/ed3d215de126 Fri, 23 Jun 2023 07:43:30 GMT 2023-07-05T14:58:29.144Z Automating a11y testing: Part 1 — Axe

Earlier this year I gave a talk at the John Lewis Partnership Tech Profession Conference 2023 on the topic of automating web accessibility (a11y) testing.

In this series I will cover the contents of this talk, kicking off with insights into the Axe suite of tools, followed by articles covering the wider a11y testing landscape: from visual regression testing for magnification and visual deficiencies, to emerging technology in the screen reader automation space.

I am not the user

Before going any futher I’m keen to express a disclaimer for this series: “I am not the user”!

I don’t have any training beyond “on the job” learnings, I am not a user of assistive technology in my day to day life outside of QA, and other than a mild astigmatism I consider myself able-bodied and neurotypical. Somewhat brings forth imposter syndrome when writing on the topic of a11y testing! Nevertheless, it’s something I’m passionate about.

Throughout this series I will be mostly focusing on the automation side of a11y testing, but it is also key to remember the importance of manual validation and exploratory testing. At the end of the day it is real people that are visiting and using your site, so it is incredibly important to ensure the quality of their experience, not just to tickbox that automation has passed. In fact, current tooling can only cover around 25% of WCAG requirements (granted an old article, but still holds true!) and although later in this series we will explore some emerging technology to start pushing that coverage higher, there are still numerous aspects of the customer experience that can only be tested by a real person.

With that all covered off, let’s have a wander through the world of automated web accessibility testing!

Current state of play

Axe

It wouldn’t be an article about web accessibility testing if the Axe suite by Deque wasn’t mentioned. Putting simply, this tooling has somewhat of a monopoly at the moment on accessibility automation, and for good reason.

“The axe family of tools help you check for digital accessibility” — https://www.deque.com/axe/

Outside of very specific, dedicated tools it has the best coverage out there, good APIs and interfaces, and although I think the technical documentation could be better (likely personal taste — it always ends up being there, just not structured how I expect), it is far from bad and is made up for by a wealth of community tutorials, articles, and repo examples.

Beyond their axe-core package Deque also maintain a number of dedicated packages for the majority of popular frameworks in the web test automation space, namely:

To give you a flavour, here is a snippet of how you might use the Playwright integration to perform static analysis on the live Waitrose Groceries site to access the accessibility of the primary navigation menu, lovingly referred to as the “mega menu”:

import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';

test('mega menu should not have a11y violations', async ({ page }) => {
  await page.goto('https://www.waitrose.com/ecom/shop/browse/groceries');

  await page.getByRole('button', { name: 'Groceries' }).click();

  await page.locator('#megamenu').waitFor();

  const accessibilityScanResults = await new AxeBuilder({ page }).include('#megamenu').analyze();

  expect(accessibilityScanResults.violations).toEqual([]);
});

Hopefully a very standard looking test — we navigate to the desired page, interact with the Groceries dropdown button, and wait for the mega menu to be visible. The Axe part is then remarkably straight-forward to get started with — we construct a new AxeBuilder, tell it to inspect the mega menu element tree, and run the analysis. Across the other framework packages the API usage is very similar.

cypress-axe

Now the eagle eyed of you will have noticed there was a popular framework missing from the list of Deque owned packages for Axe — Cypress!

Don’t worry, there is a really good community package to support Cypress as well in the form of the cypress-axe package.

The interface is a little different to how you use the other Axe packages, but personally I almost prefer the style which works quite naturally with the Cypress API style. Let’s see how we might test the accessibility of a John Lewis Search Product Listing Page (PLP):

describe('PLP Search A11y Violations', () => {
  beforeEach(() => {
    cy.visit('https://www.johnlewis.com/search?search-term=vans');

    cy.injectAxe();

    cy.configureAxe(AXE_CONFIG_OPTIONS);
  });

  it('should not have a11y violations on load', () => {
    cy.checkA11y(SELECTOR, AXE_RUN_OPTIONS);
  });
});

As is often the case with Cypress, the feedback loop from cypress-axe is also up there with clear logging explaining the issues when you are using the Cypress UI.

For example, in this failed test we can see the error description, some guidance, a URL for more information, and if we were to drill into the Nodes array it would give us pointers to the exact elements that are in violation of WCAG.

Cypress UI with a failed test for “Has no a11y violations after button click”. The test lists the error “A11Y ERROR! Heading-order on 2 Nodes” with message “1 accessibility violation was detected: expected 1 to equal 0”. This error has been pinned and in the Chrome developer tools the Console tab is open listing further details on the error, including the error id, impact, tags, description, help description and url, and the relevant DOM nodes.
cypress-axe logs in the Cypress UI

Pa11y

So what if you don’t currently use one of the aforementioned frameworks for your site?

In cases where you don’t have a setup ready to plug and play with one of the previous packages, and perhaps you’ve also not got the knowledge or resource to spend building up a suite using one of those frameworks (though these days the developer experience for setting up is pretty good!), then I would recommend taking a peek at Pa11y.

“Pa11y is your automated accessibility testing pal. It runs accessibility tests on your pages via the command line or Node.js, so you can automate your testing process.” — https://pa11y.org/

The nice thing about Pa11y is you can run it straight from the command line making it natural to use for scripting, or for simple smoke checks in CI. It also ships a Node package so you can write a quick test in JavaScript or Typescript. Because it is a “wrapper” you can also benefit from it’s dual coverage utilising both Axe and HTML Code Sniffer (htmlcs), another static analysis tool. The documentation for Pa11y is also really good.

Let’s take a look at an example setup to test the page accessibility with the mega menu open:

import pa11y from 'pa11y';

const BASE_PA11Y_CONFIGURATION = {
  runners: ['axe', 'htmlcs'],
  standard: 'WCAG2AAA',
};

async function runPa11yTest() {
  try {
    const results = await pa11y('https://www.waitrose.com/ecom/shop/browse/groceries', {
      actions: ['click element #drop-down-nav', 'wait for element #megamenu to be visible'],
      ...BASE_PA11Y_CONFIGURATION,
    });

    console.log(results);
  } catch (error) {
    console.error(error);
  }
}

runPa11yTest();

Pa11y’s actions API for perfoming customer interactions is a little restricted, but it does has you covered for basic behaviours like clicking, typing, and waiting for something to be visible. Because it uses English based instructions, it has a similar feel to using BDD testing libraries (e.g. think Gherkin syntax for Cucumber) which lowers the barrier to entry for developers or even non-developers to easily tweak or extend the scope of tests — caveated of course if English is not your first language!

Unfortunately the style of this actions API does mean the tool doesn’t sit well within other frameworks — if you’re trying to use Playwright or Cypress to first act on the page for setup it will just get ignored. What Pa11y does under the hood is use Puppeteer to spin up it’s own Chrome tab to instrument and act out the instructions and a11y testing, so anything set up outside of Pa11y’s actions array will effectively be ignored.

But if you’re performing integration testing through the likes of Jest, Vitest, Mocha, etc., i.e. a testing framework that doesn’t support it’s own browser instrumentation, this can be a nice fit to extend a suite of tests to include a11y coverage.

In summary:

  • Conveniently wraps both Axe and htmlcs
  • Really good documentation
  • Simple to use action, though somewhat restricted
  • Doesn’t play so well with others

Axe browser extension

Moving out of what some folks might strictly class as test automation, it’s worth discussing some of the other tooling in the Axe family.

John Lewis website open on the coffee table product listing page in Chrome. The developer tools pane is open on the axe DevTools tab showing the overview of an Axe scan. The report lists 4 accessibility issues in a list of expandable sections, with one for “Images must have alternative text” expanded to show details of the issue. A highlight button is toggled visually highlighting the image on the page that is missing an alt tag.
Axe browser extension running in Chrome Developer Tools

To complement the Axe packages, Deque also have a really nice Axe browser extension. Some features are premium, but the “scan all” feature meets most, if not all, needs for supplementing exploratory testing with a degree of automation, taking the heavy lifting away from manually inspecting HTML, colours, etc. to try and work out if components are compliant.

Axe Linter

There is also a VSCode plugin for an Axe Linter which can be a useful tool in proactively writing accessible code write at the early development stage — eliminating the slower feedback loop of only finding violations at your integration test stage in CI say.

A VSCode window with code for a React based Image component. The cursor is hovered over the img element JSX which doesn’t have an alt prop. The component has a red error underline and a hover tooltip which reads: “Axe Linter (image-alt): Ensures img elements have alternative text or a role of none or presentation (dequeuniversity/image-alt)”.

Lighthouse

Lastly something that hopefully might be familiar to most readers!

If you’ve ever used the Accessibility reporting feature of Google Lighthouse, it uses Axe under the hood to drive the tests and provide the nicely displayed violation information.

This is not just the case for the Chrome DevTools instance of Lighthouse — the browser extension, CLI, and Node package all make use of Axe for the accessibility score and reporting features.

Waitrose Groceries favourites page open in Chrome using iOS mobile simulation. The DevTools panel is open on the Lighthouse tab displaying an Accessibility score for the page. WCAG violations are listed with descriptions of the issue, and references to the impacted elements with screenshots highlighting the elements.
Google Chrome Lighthouse DevTools

There are also some nice packages that wrap Lighthouse for easy use in test automation — for example, the cypress-audit package for Cypress worthy of a mention for it’s easy to use cy.lighthouse() command which can also be used for performance budget tests (it also has a cy.pa11y() command so you can kick off Pa11y tests if you prefer that interface instead).

Axe lessons

Having realed off and recommended a number of Axe tools, it’s worth pointing out some of the gotcha’s that I’ve experienced with them over the years.

React Axe issues

A big lesson for me has been around @axe-core/react. Initially I was very much in favour in the idea of having a tool that can instrument React and report issues to the console while running apps locally in development.

However… for any application of any size this package can massive tank performance which can make local development painful and slow — especially if you are running a hot module reloading (HMR) setup where every small tweak triggers a hot reload and a potential new Axe scan of the page. If you are using it currently and ever wonder why animations are super laggy and page updates look jarring, then React Axe is a real possibility as to why. Obviously this is my experience working on large applications, mileage may vary!

My personal recommendation is to scrap React Axe and instead reach for the Axe browser extension instead — you don’t get “live” reports to the console, but the flow is fairly natural and is a definite improvement for developer experience. The UI and level of detail you get on the extension is also far superior to the red text you get in the console scattered amongst everything else that is getting logged.

Infallible?

It’s also worth calling out that Axe isn’t perfect — it can occasionally have quite annoying false positives.

The most common case I’ve found is in it’s reporting of colour contrast, specifically when doing anything complex with either nested elements or using pseudo-elements.

For example, imagine you have a setup (admittedly contrived, but funnily enough I have experienced this in a side menu implementation!) where there is an element with a green background and then a number of nested transparent that cover the parent element. Above these is then say an absolutely positioned element, or CSS “Z translated” element, or perhaps an element from an entirely different part of the DOM positioned also above the green element and it’s covering children. Say this too has some nested children — the top layer of which has some white text.

The white text is above the green background, so this is WCAG AA compliant but the element containing the text is sufficiently removed from the background element that Axe fails to recognise which elements (and colours) are involved in the contrast comparison and fails with a violation.

The trick here is to keep it simple — if you need to ensure contrast just apply the appropriate background (or equivalent) to the element with the text. Yes, this is somewhat fixing a non-issue due to tooling limitations, but it is slightly less fragile to changes in your page markup implementation — in future someone might refactor to reduce the complexity of this component, and having the element satisfy the necessarily colour contrast requirements itself rather than relying on some other element will save future rework getting things compliant again in such a refactor. As the principle says, “things that move together should sit together”.

There is also another lesson here — if you’re seeing false positives reported from Axe you’re probably doing something off in your markup. There are few scenarios that warrant such a complex element layout so Axe failing is probably a good hint that you’re in need of simplifying your HTML. Even if you are able to successfully satisfy the Axe requirement, it is very likely with such a layout that you will have potential issues with other accessibility requirements —what about users who make use of 2x zoom, or assistive technologies like magnifiers and screen readers? There is a real risk with something like above behaving poorly for some of these tools, e.g. screen readers are known for not handling non-semantic deeply nested markup very well.

Friction in WIP projects

Unless you’re in the rare position of starting from scratch with a greenfield project, the chances are you may well be pulling something like Axe into an existing project for a site that isn’t all that accessible at the moment, as a means to start improving the situation.

The thing to be aware of here is Axe’s limitations on ignoring violations, a capability that is often needed as a practical measure when you are looking to progressively improve a site’s accessibility which has a number of existing violations.

Of course ideally we don’t want to be ignoring anything, but a nice flow when working on such a codebase is to always create a ticket, and then add ignore rules named after or commented with the ticket reference and start ticking these off.

The frustrating thing with Axe ignores is that you can’t ignore a specific rule for a specific element. Unfortunately you can either ignore all rules for an element, or you can ignore all cases of a rule. The consequence is that often you will be over-ignoring elements or rules until you have cleared the decks of most or all violations, meaning until then you’re not really getting the protection from regression that you might hope for — folks adding new code may well introduce further violations of a similar type and Axe starts ignoring these as well!

What I recommend here is set up your tests from the start with sensible per test configuration and avoid having a single global configuration (though for DRY, it might be some common config is centralised). This way you can narrow the exclusions just for specific tests, e.g. say the contrast is only inaccessible when you toggle a button, you don’t want to ignore that button or the contrast rule for all other tests and scenarios in your suite!

Another technique that I’ve seen employed to reasonable effect is to introduce attributes to elements that you want to ignore, for example data-axe-ignore="true", and use this as a way to target elements that you need to temporarily ignore (through a selector such as [data-axe-ignore="true"]) in order to get CI up and running with Axe tests. Once you fix the violation, just remember to remove the data attribute! This does have the downside of polluting production markup unless you have steps to purge the attributes, so there are trade-offs here.

Closing notes

In this article I’ve covered off the majority of the Axe suite, if you want to find out more or explore some of Deque’s other offerings check out their site at https://www.deque.com/axe/.

In part 2 of this series I will cover a number of non-Axe tools and techniques for a11y test automation. See you there! 👋

Hi, my name is Craig Morten. I am a senior product engineer at the John Lewis Partnership. When I’m not hunched over my laptop I can be found drinking excessive amounts of tea or running around in circles at my local athletics track.

The John Lewis Partnership are hiring across a range of roles. If you love web development and are excited by accessibility, performance, SEO, and all the other awesome tech in web, we would love to hear from you! See our open positions here.

Automating a11y testing: Part 1 — Axe was originally published in John Lewis Partnership Software Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

]]> <![CDATA[Reducing a cryptic memory leak]]> https://medium.com/@craigmorten/reducing-a-cryptic-memory-leak-in-production-e36384208ea5?source=rss-c6b59d857cb7------2 https://medium.com/p/e36384208ea5 Sat, 18 Mar 2023 15:59:21 GMT 2024-01-27T14:53:27.836Z “Reducing” a cryptic memory leak in Production

I recently came across an interesting memory leak scenario which caught me off-guard, let’s see if you can spot the issue before the reveal at the end?

Below is some simplified replica code:

const express = require("express");
const { getSponsored, getRecommendations, getProducts } = require("./fetchers");

const app = express();

const baseResponse = {
  filters: [],
  products: [],
  subCategories: [],
};

function mergeResponses(responses) {
  return responses.reduce((finalResponse, response) => {
    finalResponse.filters.push(...response.filters);
    finalResponse.products.push(...response.products);
    finalResponse.subCategories.push(...response.subCategories);

    return finalResponse;
  }, baseResponse);
}

app.use("/products", async (req, res) => {
  const responses = await Promise.all([
    getSponsored(),
    getRecommendations(req),
    getProducts(),
  ]);

  res.send(mergeResponses(responses));
});

app.listen(3000);

The functionality is relatively straight-forward — we have a /products endpoint which is getting some sponsored, recommended, and normal products, merging parts of the results, and returning this aggregation as the response.

Now the use of Array.prototype.reduce() for the response merge is curious — we’re concatenating arrays so a simple for loop or .forEach() feels more appropriate, but hey for small responses at reasonably low RPS this is micro-optimisation, nothing to get too hung up on!

MacBook with VSCode open on an Express JS file.
Photo by Clément Hélardot on Unsplash

But yet running something similar in a production like environment it was apparent that something was off… monitoring showed that old heap size was growing linearly with the number of requests — we had a memory leak!

The memory heap is divided into two major spaces: the New space where new objects are allocated, and the Old space where older objects are moved to after surviving for some time having not been garbage collected.

Debugging a memory leak

A straight-forward way to start an investigation into a Node memory leak when you can run the project locally (debugging in production is a different matter!) is to use the Node inspector, this can be invoked easily through a flag when starting your Node application:

$ node --inspect index.js

Debugger listening on ws://127.0.0.1:9229/2a7f05cd-96c8-4f6d-8c5f-adea987ab63b
For help, see: https://nodejs.org/en/docs/inspector

This will prompt the Node process to listen for a debugging client which, once connected, allows you to instrument your application with breakpoints as well as resource metrics and analysis for CPU and memory.

There a few options for debuggers — major IDEs like Visual Studio, Visual Studio Code, and JetBrains Webstorm all support the protocol making for easy debugging from within your editor. Here I’ll demonstrate an alternative, using Chrome DevTools.

Once your Node process is running with the inspector, open up Chrome and navigate to chrome://inspect in the URL address bar.

Chrome browser open on the chrome://inspect page, listing a target for Node v14.20.1 for index.js and an option to inspect.
Chrome Inspect Page

Here under the “Remote Target” section we can click on the “inspect” link to starting interrogating our index.js Node process.

Chrome DevTools window open on the Memory tab showing options for selecting a profiling type.
Chrome DevTools Memory Tab

Clicking this link pops open a new Chrome DevTools tab which is very similar to what you would get while debugging a page in the browser.

On the Memory tab we can click the “Take snapshot” button while the “Heap snapshot” option is selected to record a snapshot in time of the memory heap. This will pop up on the left sidebar and generally will automatically open up.

Chrome DevTools heap snapshot listing rows of Constructors.
Chrome DevTools Memory Heap Snapshot

From this snapshot we can see all of the “constructors” (think primitives or objects) with columns for the distance from the window object, the size of each object in bytes, and the retained size of the object in bytes (the size of the object and the graph it retains).

This doesn’t yet give us enough information to easily assess where a memory leak might be, but can be interesting to peruse to understand the shape of your application’s memory.

Now in our case the memory leak appeared to be correlated to the amount of traffic the server was receiving, so to simulate that we can use tools like autocannon to very easily generate some base level load on the server.

Here was run one of three:

$ npx autocannon -c 100 http://localhost:3000/products 
  
Running 10s test @ http://localhost:3000/products
100 connections


┌─────────┬────────┬─────────┬─────────┬─────────┬───────────┬───────────┬─────────┐
│ Stat    │ 2.5%   │ 50%     │ 97.5%   │ 99%     │ Avg       │ Stdev     │ Max     │
├─────────┼────────┼─────────┼─────────┼─────────┼───────────┼───────────┼─────────┤
│ Latency │ 158 ms │ 1232 ms │ 2860 ms │ 3440 ms │ 1296.6 ms │ 786.21 ms │ 5350 ms │
└─────────┴────────┴─────────┴─────────┴─────────┴───────────┴───────────┴─────────┘
┌───────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┐
│ Stat      │ 1%      │ 2.5%    │ 50%     │ 97.5%   │ Avg     │ Stdev   │ Min     │
├───────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Req/Sec   │ 27      │ 27      │ 46      │ 147     │ 61.8    │ 35.14   │ 27      │
├───────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Bytes/Sec │ 34.4 MB │ 34.4 MB │ 60.6 MB │ 80.7 MB │ 60.7 MB │ 13.3 MB │ 34.4 MB │
└───────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┘

Req/Bytes counts sampled once per second.
# of samples: 10

718 requests in 10.09s, 607 MB read

… and run three of three:

$ npx autocannon -c 100 http://localhost:3000/products

Running 10s test @ http://localhost:3000/products
100 connections


┌─────────┬─────────┬─────────┬─────────┬─────────┬────────────┬────────────┬─────────┐
│ Stat    │ 2.5%    │ 50%     │ 97.5%   │ 99%     │ Avg        │ Stdev      │ Max     │
├─────────┼─────────┼─────────┼─────────┼─────────┼────────────┼────────────┼─────────┤
│ Latency │ 1554 ms │ 5400 ms │ 9049 ms │ 9189 ms │ 5389.29 ms │ 2438.84 ms │ 9314 ms │
└─────────┴─────────┴─────────┴─────────┴─────────┴────────────┴────────────┴─────────┘
┌───────────┬─────┬──────┬─────────┬────────┬─────────┬─────────┬─────────┐
│ Stat      │ 1%  │ 2.5% │ 50%     │ 97.5%  │ Avg     │ Stdev   │ Min     │
├───────────┼─────┼──────┼─────────┼────────┼─────────┼─────────┼─────────┤
│ Req/Sec   │ 0   │ 0    │ 6       │ 36     │ 10.6    │ 11.62   │ 6       │
├───────────┼─────┼──────┼─────────┼────────┼─────────┼─────────┼─────────┤
│ Bytes/Sec │ 0 B │ 0 B  │ 22.4 MB │ 128 MB │ 38.1 MB │ 41.5 MB │ 22.4 MB │
└───────────┴─────┴──────┴─────────┴────────┴─────────┴─────────┴─────────┘

Req/Bytes counts sampled once per second.
# of samples: 10

232 requests in 10.09s, 381 MB read
26 errors (26 timeouts)

We can see here that something is very definitely wrong — after only 3 batches of 10 second load periods serving 100 connections we’ve gone from an average latency of 1296.6 ms to 5389.29 ms, a degradation of over 4 seconds!

Let’s take another heap snapshot using the “record” button at the top left of the Chrome DevTools Memory tab.

Chrome DevTools window open on the Memory tab with the record button highlighted on the top left of the Memory tab’s control panel. A second heap snapshot is open.
Chrome DevTools Memory Heap Snapshot 2

We can see that there certainly appears to be a lot more memory in play — the total snapshot size (listed under the snapshot name on the left sidebar) has increased from 5.6 MB to 24.7 MB and there are some large numbers listed for shallow and retained size.

However hunting blind in any one snapshot doesn’t get you very far typically, where the DevTools comes into it’s own is with the comparison options.

The last option in the Memory Tab’s control panel is a dropdown which will state “All objects” by default. Clicking on this dropdown gives you a number of options to compare between snapshots.

Chrome DevTools window open on a heap snapshot, with the comparison dropdown open in the Memory Tab control panel. Listing options: All objects, Objects allocated before Snapshot 1, Objects allocated between Snapshot 1 and Snapshot 2.
Chrome DevTools Memory Heap Comparison Dropdown

Here we select the “Objects allocated between Snapshot 1 and Snapshot 2” option to show us the delta between the two snapshots.

Chrome DevTools window showing the difference between Snapshot 1 and Snapshot 2
Chrome DevTools Memory Heap Comparison View

This immediately reduces the number of constructors in play when compared with the long lists for the individual snapshots — anything that hasn’t changed between the snapshots hasn’t been listed, removing a lot of the noise from the view.

Now comes the interesting part — finding the source of the leak! We start by sorting the list by retained size (typically the default) and navigating down the list of constructors one by one. Here we expand the (string) options first and take a look:

Chrome DevTools window with the strings memory expanded.
Chrome DevTools Memory Heap strings

There’s nothing of interest here — we can tell from the small shallow and retained sizes of each of the strings listed that the issue doesn’t lie here. Although the combined size of each string does make for the largest sized collection, the size associated with each string is very small.

Intuitively we’re also not expecting a string to be the primary cause of our problem — nowhere in our logic are we growing a string (perhaps if we were doing some sort of string concatination, but we’re not). It’s apparent that a lot of these strings are likely long-lived constants, we likely took our first snapshot too early — if we were to take a third snapshot they would like be removed from the comparison view now the server has had some requests.

Moving onto the second constructor we can see something a little more interesting:

Chrome DevTools window with the arrays memory expanded.
Chrome DevTools Memory Heap Comparison arrays

Here the shallow and retained sizes for the top three children are particularly large considering this is a delta!

Clicking on the first (object elements)[] row we can populate the “Retainers” bottom panel with detailed information on the object in question. This immediately points to a suspect:

Chrome DevTools window with the arrays memory expanded and the first object selected. The Retainers bottom panel is open showing tree graph view of object relationships and their memory allocation.
Chrome DevTools Memory Heap Retainers View

In the bottom panel we can see that the massive retained size has been associated with the elements in a products array that exists on a baseResponse object used by the Express middleware stack! If we click on the second and third child we can see that these are for subCategories and filters respectively.

Somehow these arrays on our baseResponse object are growing in size after requests. This must mean that they are being mutated with more and more values! In fact if we expand the (object elements)[] children we can see the exact values that are populating our arrays.

So our leak is something causing the baseResponse arrays to grow on a per request basis… lets go have another look.

Want to learn more about debugging memory with Node? Checkout the Node Memory Diagnostics tutorial page.

When does a reduce grow?

Let’s take another look at our mergeResponses() method:

const baseResponse = {
  filters: [],
  products: [],
  subCategories: [],
};

function mergeResponses(responses) {
  return responses.reduce((finalResponse, response) => {
    finalResponse.filters.push(...response.filters);
    finalResponse.products.push(...response.products);
    finalResponse.subCategories.push(...response.subCategories);

    return finalResponse;
  }, baseResponse);
}

We’re reducing over the array of responses, accumulating the associated filters, products, and subCategories and return the result. And in doing so somehow the baseResponse is being mutated?

Well exactly that — TIL!

Turns out that although Array.prototype.reduce() does not mutate the underlying array it does mutate the initialValue if it is provided. In many cases where you might provide a value directly like a number or an empty object this is fine, but if you create a bootstrap object like we have here then it will get mutated by each call, and re-used every time!

This isn’t mentioned anywhere on MDN or similar from what I can find, so is a bit of a gotcha — though I suspect the number of times this pattern is used is likely (hopefully) small.

So how do we fix this issue?

Well one option is to move the baseResponse into the mergeResponses() method so that it is no longer in top level scope and can be garbage collected once the request has been responded to. I.e.

function mergeResponses(responses) {
  const baseResponse = {
    filters: [],
    products: [],
    subCategories: [],
  };

  return responses.reduce((finalResponse, response) => {
    finalResponse.filters.push(...response.filters);
    finalResponse.products.push(...response.products);
    finalResponse.subCategories.push(...response.subCategories);

    return finalResponse;
  }, baseResponse);
}

Or as we opted for, move away from a reduce when we’re fundamentally not changing object shape:

function mergeResponses(responses) {
  const baseResponse = {
    filters: [],
    products: [],
    subCategories: [],
  };

  responses.forEach((response) => {
    baseResponse.filters.push(...response.filters);
    baseResponse.products.push(...response.products);
    baseResponse.subCategories.push(...response.subCategories);
  });

  return baseResponse;
}

So did you spot the memory leak before the big reveal?

Have you got the same code in your apps? 😱

Have you faced a memory leak recently?

Let me know in the comments along with your thoughts, comments, and queries! Till next time folks 🚀

]]> <![CDATA[A11y Unlocked: Screen-Reader Automation Tests]]> https://medium.com/@craigmorten/a11y-unlocked-screen-reader-automation-tests-788b5c6bac8d?source=rss-c6b59d857cb7------2 https://medium.com/p/788b5c6bac8d Sun, 16 Oct 2022 17:31:50 GMT 2024-01-27T14:53:51.390Z http://creativecommons.org/licenses/by/4.0/ In the past year since my last post A11y Testing: Automating ScreenReaders on the topic of automating screen readers, we’re now in a better place with progress made in standards and tooling.

It is exciting to see the W3C ARIA-AT Community Group formed with a mission to improve screen reader interoperability by building out a suite of specifications, test standards, and test automation to mature the screen reader ecosystem towards first-class tooling.

“Enabling AT interoperability is a large, ongoing endeavor that requires industry-wide collaboration and support. The W3C ARIA-AT Community Group is focusing on a stable and mature test suite for five screen readers by the end of 2023. In the future, we plan to test additional screen readers and other kinds of assistive technologies with a broader set of web design patterns and test material.”
Source: https://aria-at.w3.org/

While the standards community slowly work towards their long term goal, I am pleased to share some screen reader automation tooling which you can use today! 🚀

Guidepup for playwright

In this tutorial we’re going to make use of the package @guidepup/playwright to write e2e tests for testing the accessibility of pages for people who use the VoiceOver screen reader for MacOS.

Logo of guidedog puppy with text: VoiceOver Automation With Guidepup

The package provides a reliable set of APIs to automate your screen reader a11y workflows when using Playwright. It does this by providing both Playwright configuration as well as a custom Playwright test instance that exposes a voiceOver object for controlling VoiceOver with.

Getting setup

Let’s create a new project:

npm init

Install the @guidepup/playwright package as a dependency:

npm install @guidepup/playwright

We’ll also need Playwright as the test runner for our browser:

npm install @playwright/test

Finally we’ll want to test against Safari using Playwright, so let’s use their tool get grab the browser:

npx playwright install webkit

We now have all the dependencies we need! 🚀

A touch of config

Create a new playwright.config.js file:

import { voConfig } from "@guidepup/playwright";

const config: PlaywrightTestConfig = {
  ...voConfig,

  // Your custom config ...
};

export default config;

Here we’re pulling in the minimum recommended Playwright config for using Guidepup — you can check it out in the source code. This sets the number of workers to 1 and sets Playwright to use "headed" mode because when we're using VoiceOver we can only control one browser at a time (and ideally it's visible!).

We should now be completely set to get going with our first test! 🎉

Writing your first screen-reader test

Create a new file voiceover.spec.js and copy in the following contents:

// We use a special test instance from the Guidepup package
// that gives us access to VoiceOver!
import { voTest as test } from "@guidepup/playwright";
import { expect } from "@playwright/test";

// The test setup is exactly the same as normal for
// Playwright, expect we now get a `voiceOver` object as well
// as the `page` object!
test.describe("Playwright VoiceOver", () => {
  test("I can navigate the Guidepup Github page", async ({
    page,
    voiceOver,
  }) => {
    // Let's navigate to Guidepup GitHub page and wait for
    // page to be ready, nothing special here!
    await page.goto("https://github.com/guidepup/guidepup", {
      waitUntil: "domcontentloaded",
    });
    await expect(
      page.locator('header[role="banner"]')
    ).toBeVisible();

    // This is where things start to get awesome! Let's tell
    // VoiceOver to interact with the main page content, just 
    // the same as you would when use VoiceOver normally.
    await voiceOver.interact();

    // Let's do something a lil more exciting - move across
    // the page's headings until we reach the main Guidepup
    // repo title in the README using VoiceOver!
    while ((await voiceOver.itemText()) !== "Guidepup heading level 1") {
      await voiceOver.perform(
        voiceOver.keyboard.commands.findNextHeading
      );
    }
});

Check out the comments in the code for what will happen in each part — pretty awesome huh? 🙌

Last checks before we launch

In order to automate screen-readers a few OS level settings need to be sorted first!

You can head over to the Guidepup environment setup docs for more details, but I recommend trying out the @guidepup/setup package which is designed to get you setup no faff!

npx @guidepup/setup

You’re now ready to rock! 🤘

Running your first screen-reader test

No special commands needed here, just the same as with any other Playwright test! Fire off the command then hold tight — we don’t want to interact with the machine while the test is running as it might move VoiceOver’s focus!

npx playwright test
Gif of Playwright controlled Safari browser being driven with VoiceOver. Announcements read: “heading level 2, Latest commit”, “heading level 2, Git stats”, “You are currently on a heading level 2.”, “heading level 2, Files”, “You are currently on a heading level 2.”, “heading level 2, link, README.md”, “heading level 1, Guidepup”
A11y test for a GitHub page on Safari using Guidepup to drive VoiceOver automation.

What’s next?

To find out more there’s a whole load of reading, modules, and docs — check some of these out:

Time to try out some e2e screen-reader tests?

Let me know your thoughts, questions, and opinions in the comments!

Till next time folks 👋

Originally published at https://dev.to on October 16, 2022.

]]> <![CDATA[Testing Web Vitals With Cypress]]> https://medium.com/@craigmorten/testing-web-vitals-with-cypress-60d927546062?source=rss-c6b59d857cb7------2 https://medium.com/p/60d927546062 Sun, 03 Apr 2022 10:49:14 GMT 2024-01-27T14:54:05.561Z http://creativecommons.org/licenses/by/4.0/ It is well understood that performance is a critical consideration for any website which can have far reaching impacts on everything from customer satisfaction, SEO, and ultimately your bottom line. You cannot determine the success of performance work without the ability measure the results and compare to performance budgets — this calls for testing infrastructure to make sure you have the necessary visibility on metrics… introducing cypress-web-vitals.

cypress-web-vitals allows you to test against the Google Web Vital signals within your Cypress workflows through a new cy.vitals() custom command.

Web Vitals is an initiative by Google to provide unified guidance for quality signals that are essential to delivering a great user experience on the web.
Reference: https://web.dev/vitals/

Getting started

In your project, install the dependencies:

npm install --save-dev cypress-web-vitals cypress-real-events
Note: cypress-web-vitals currently makes use of cypress-real-events to click the page to calculate first input delay. Hence it is needed as a peer-dependency.

Within you support commands file (normally cypress/support/commands.js), add this one liner to get setup:

import "cypress-web-vitals";

And now you’re set to get going with Web Vital performance budget tests in your Cypress workflow! 🎉

Add you first test like so:

describe("Web Vitals", () => {
  it("should pass the meet Google's 'Good' thresholds", () => {
    cy.vitals({ url: "https://www.google.com/" });
  });
});

You are now set up to test against all of the Web Vitals using Google’s “Good” threshold values:

Customise your tests

You can further customise your tests through additional optional configuration which is passed to the cy.vitals(webVitalsConfig) call:

  • Optional url: string - The url to audit. If not provided you will need to have called cy.visit(url) prior to the command.
  • Optional firstInputSelector: string - The element to click for capturing FID. The first matching element is used. Default: "body".
  • Optional thresholds: object - The thresholds to audit the Web Vitals against. If not provided Google's "Good" thresholds will be used. If provided, any missing Web Vitals signals will not be audited.
  • Optional vitalsReportedTimeout: number - Time in ms to wait for Web Vitals to be reported before failing. Default: 10000.

For example:

// Use the `main` element for clicking to capture the FID.
cy.vitals({ firstInputSelector: "main" });

// Test the page against against a CLS threshold of 0.2.
cy.vitals({ thresholds: { cls: 0.2 } });

For more details on usage refer to the API docs.

How does it work?

  1. The url is visited with the HTML response intercepted and modified by Cypress to include the web-vitals script and some code to record the Web Vitals values.
  2. Several elements (if exist) starting with the provided element (based on firstInputSelector) are then clicked in quick succession to simulate a user clicking on the page. Note: if choosing a custom element, don't pick something that will navigate away from the page otherwise the plugin will fail to capture the Web Vitals metrics.
  3. The audit then waits for the page load event to allow for the values of LCP and CLS to settle; which are subject to change as different parts of the page load into view.
  4. Next the audit simulates a page visibility state change which is required for the CLS Web Vital to be reported.
  5. The audit then attempts to wait for any outstanding Web Vitals to be reported for which thresholds have been provided.
  6. Finally the Web Vitals values are compared against the thresholds, logging successful results and throwing an error for any unsuccessful signals. Note: if the audit was unable to record a Web Vital then it is logged, but the test will not fail.

Testing sites in the wild

Here are some example test runs against FAANG homepages to see cypress-web-vitals in action:

Facebook

cy.vitals({ url: "https://www.facebook.com/" });
Cypress Web Vitals test against the Facebook landing page. All metrics below Google’s “Good” thresholds.

Amazon

cy.vitals({ url: "https://www.amazon.com/" });
Cypress Web Vitals test against the Amazon home page. All metrics below Google’s “Good” thresholds.

Netflix

cy.vitals({
  url: "https://www.netflix.com/gb/",
  firstInputSelector: ".our-story-card-title",
});
Cypress Web Vitals test against the Netflix landing page. All metrics below Google’s “Good” thresholds.

For Netflix we have had to introduce a custom element choice for the simulated “first click”. Even though first input delay can be measured when in cases where there is no event listener registered to the element, there are scenarios where clicking the body doesn't cut it. Some good examples of elements that will reliably trigger an FID metric are:

  • Text fields, checkboxes, and radio buttons (<input>, <textarea>)
  • Select dropdowns (<select>)
  • Links (<a>)

In order to ensure the Web Vitals can be tested against, it is best to try find an element that reliably triggers an FID, but won’t navigate away from the page (perhaps avoid <a>!).

Google

cy.vitals({ url: "https://www.google.com/" });
Cypress Web Vitals test against the Google home page. All metrics below Google’s “Good” thresholds.

Wrap-up

Using any awesome performance testing tooling lately? Tried out cypress-web-vitals on your site and have results to share? Got any thoughts, queries, or questions? Leave a comment!

That’s all folks! 🚀

Originally published at https://dev.to on April 3, 2022.

]]>