Add visual regression testing to your CI pipeline for React Native apps.
This is a high-level overview of how you can build and integrate visual regression tests into a React Native application pipeline. It utilises the Detox screenshot API and resemble.js library for image comparison.
What is Visual regression testing?
It’s the process of generating and comparing images to showcase differences. It is an essential part of maintaining the visual integrity of an app. We do this in the following way: Capture a set of base screenshots as a reference point for our app’s user interface (UI). Then, as we add new features, capture an updated (current) set of screenshots. Finally, we compare them to identify and highlight discrepancies.
Why bother with it?
There are several reasons why visual regression testing can be useful:
- Consistency: It ensures your application remains consistent over time by flagging UI differences. This is important for maintaining a cohesive and professional image for your app. It also eases refactoring components, assuring changes haven’t introduced a visual change.
- User experience: Visual changes can impact user experience significantly. For example, if a button was easy to locate is now hidden, it could cause frustration for users.
- Brand image: The visual appearance of your app is often closely tied to the brand image. Visual regression testing helps ensure that changes to your app do not damage it.
- Reduce development time: With visual regression testing, you can catch visual issues fast. This can save time and effort in the long run by reducing the need for manual testing.
In summary, visual regression testing can help ensure your app’s UI stays consistent. It also saves time and money by catching issues early on.
Let’s look at how this can be achieved in your continuous integration (CI) pipeline.
Which visual regression service is best?
There are a few visual regression testing services that are will-suited for react-native:
- Applitools: This is a popular choice for React Native developers. It supports automated visual testing of both web and mobile apps. It offers a range of features, including layout testing and integration with popular test frameworks like Jest and Detox.
- Percy: Integrates with React Native projects and automates visual regression testing. It uses a plugin for Jest that allows you to take screenshots of your React Native app. Using this can quickly and easily detect visual changes.
- Detox by Wix: This is an end-to-end testing library for React Native apps. It supports integration with popular test runners like Jest. It also includes a built-in screenshot utility that we can use for visual regression testing.
The best choice for your React Native project will depend on specific requirements and available resources. Each service has its strengths. Consider them carefully to find the one that best meets your needs.
Our solution — Detox screenshot API
We use Detox for our end-to-end (e2e) tests and utilise their built-in screenshot API for our visual regression testing solution. This option made sense to us because we were already using Detox and didn’t need all the features that the other services provided.
In each PR we run our e2e tests for Android and iOS taking screenshots of our app. Then use those screenshots and compare them against our base screenshots to generate diff screenshots. This worked nicely with not too much work. But, the issue was displaying the diff images to engineers. We first saved them to CircleCI’s artifacts, which meant they were visible in a tab inside CircleCI. While this surfaced the diff images, it didn’t surface them high enough and you had to click on them individually one at a time. For this reason, we used GitHub REST API and sent those images as a comment, providing a great visual overview of our apps.
A caveat to this solution — Since we’re taking screenshots during our e2e tests, it limits us in what we can screenshot. E2e tests should test the happy path and not every possible route a user can take along with all possible component states. Thus, we wouldn’t get 100% visual test coverage. If we did it would lead to extremely long tests.
Here are the jobs we created along with others to make this happen.
High-level view
A high-level overview looking at all the CI jobs involved when we create a PR on GitHub.
1). Pull base screenshots 🔽
Base images are up-to-date screenshots of our app on our master branch. When we first set this up, we have to manually create them (Refer to step 2 if you’re setting up for the first time). After that, we programmatically update our base images when we merge PRs.
We use an orb called circleci/aws-s3 orb to copy screenshots from our s3 bucket. Once we’ve copied them we persist them to our workspace in CircleCI. Meaning any job afterwards that attaches the workspace has access to the base images. See the CircleCI job below.
// Copy base images from s3 bucket to "e2e/tmp/base/"
- aws-s3/copy:
aws-access-key-id: AWS_ACCESS_KEY_ID
aws-secret-access-key: AWS_SECRET_ACCESS_KEY
aws-region: AWS_DEFAULT_REGION
from: $AWS_S3_SCREENSHOT_BUCKET/base/
to: e2e/base/
arguments: |
--recursive
// Persist images to workspace
- persist_to_workspace:
root: .
paths:
- e2e/base/*2). Run e2e Detox tests (Take screenshots) 📱
We’re piggybacking off our e2e tests to take screenshots while running through our e2e tests. This helps to reduce our pipeline time, rather than having to run a separate job. Here’s the API Detox offers to take screenshots. After our e2e tests finish in Detox they’re stored in a hidden folder called artifacts. From there we can use those images in the artifacts folders to create our current screenshots that we’ll use to compare to our base screenshots.
3) Copy e2e screenshots from artifacts to the current folder 🚋
This job copies and moves the newly created screenshots to a folder e2e/tmp/current. It makes it easier for our next job.
4) Compare screenshots (diff using resemble.js)
This job compares our e2e/tmp/base and e2e/tmp/current screenshots outputting e2e/tmp/diff screenshots. We use resemble.js for the comparison that generates images like so:
const compareImages = require('resemblejs/compareImages')
const OPTIONS = {
output: {
errorColor: {
red: 255,
green: 0,
blue: 255,
},
errorType: 'movement',
transparency: 0.3,
largeImageThreshold: 1200,
useCrossOrigin: false,
outputDiff: true,
},
scaleToSameSize: true,
ignore: 'antialiasing',
}
const getDiff = async (imgName) => {
const data = await compareImages(
await readFile(`base/${imgName}`),
await readFile(`current/${imgName}`),
OPTIONS
)
await writeFile(`${DIR_OUTPUT}/${imgName}`, data.getBuffer())
}
const compareAllImages = async () =>
Promise.all(readdirSync(DIR_CURRENT).map(getDiff))
await compareAllImages()5). Push diff and current screenshots to an S3 bucket 🗃
We push the diff and current screenshots to an S3 bucket so that we can create absolute image URLs to display the diff images as a comment in GitHub. We use the current screenshots to update the base screenshots once we merge the PR (See step 7).
Note — we tried converting images into base64 but GitHub had issues displaying them.
6). Display diff screenshots as a PR comment 🗣
We use the GitHub REST API to send the screenshots to the GitHub PR. Here is the GitHub API to create an issue comment.
curl \
-X POST \
-H "Accept: application/vnd.github+json" \
-H "Authorization: Bearer <YOUR-TOKEN>" \
https://api.github.com/repos/<OWNER>/<REPO>/issues/<PULL_NUMBER>/comments \
-d "$STRINGIFY_BODY"7). Promote base images and clean up on merge 🏅
We create a GitHub workflow that triggers when we merge the PR. The action promotes our current images and saves over our base images. At the same time, we delete the PR images stored on the S3 bucket.
An issue we came across was when more than one PR closes at the same time. How do we avoid promoting/copying images at the same time? Luckily GitHub actions offer a concurrency key. This ensures that only a single job or workflow using the same concurrency group will run. So in our case, if we merge several PRs simultaneously, whichever was first will run. The others will pause until it’s finished. See below a snapshot of the GitHub action code.
name: On Pull Request close
on:
pull_request:
types: [closed]
# Ensure that only a single job or workflow using the same concurrency group will run at a time.
# We don't want any conflicts when multiple actions are trying to promote base images
concurrency: promote-base-images-id
jobs:
pr_closed_cleanup:
runs-on: ubuntu-latest
if: github.event.pull_request.merged
...The result
All our PRs receive 2 GitHub comments showing visual regression images. This is for both iOS and Android emulators providing us with extra confidence. It also helps give our engineers a high-level, visual overview of our app in an automated way. Thus, improving our confidence and quality assurance of our app.
Note on tracking base images
We can take 3 routes here when tracking base images.
- Store directly in the repository — Saving base images within your project proved difficult. We still had to store the current images somewhere to display them in GitHub. The next issue was promoting/updating them on our master branch. Our master branch has protection rules enabled which make merging into it difficult. We didn’t want to create a new PR each time to update the base folder. We’d have double the PRs!
- Store in GitHub artifacts — You’d store the artifacts on master commits then use a script to pull down the base images. These can be pulled down whenever needed but uses your GitHub credits. This is doable but just comes down to pricing.
- Store in an S3 bucket — Very cheap plus we’re removing them as soon as we merge a PR. This was also simple to set up and to pull and push images because there’s an orb to help.
In summary, we stored our images in an S3 bucket because it was cheap and easy to do. But, if you like the idea of it all in one place and don’t mind the credit usage then I’d go for storing them in GitHub artifacts.
