Visual Regression testing with Cypress.io and cypress-image-snapshot

Published in

Norwich Node User Group

5 min readMar 4, 2019

This article should help you decide as to whether you need to add visual regression testing to your toolkit and how to do it. This publication is timed to coincide with the presentation taking place with the Norwich Node User Group.

I’m going to be using Cypress.io as the test framework, it is simple to install, use and gives you fast feedback with hot reloading the tests you create, as you write them. There are existing articles on why Cypress is great, like this one right here on Medium, authored by Evelyn Chan.

What is visual regression testing?

Visual regression testing is the ability to cross-reference snapshots of a product, highlighting pixel differences — returning fast feedback on the current state of the screen.

The reason it's important!

Browser automation covers the functionality of your application, not what it looks like! How can you be sure the styling is as it should be? …With image diffing!

Cypress

At the time of writing is yet to implement visual regression testing, it is, however, a current open proposal on GitHub.

To fill this gap, we are turning to the node package cypress-image-snapshot.

Installation & Configuration

Pre-requisites here are node.js, npx and an IDE of your choice… I prefer vsCode.

Open your project, and run the following commands

npm install cypressnpm install cypress-image-snapshot

Make changes to the following files:

// package.json

{
…
"scripts": {
  "cy:open": "cypress open"
}
…
}

// cypress.json

{
“baseUrl”: “<websiteUnderTest>”,
 “video”: false
}

Video false here is optional to save on space, it has no bearing on the outcome of the tests.

// cypress/plugins/index.js

const {
 addMatchImageSnapshotPlugin,
} = require('cypress-image-snapshot/plugin');
module.exports = (on, config) => {
  addMatchImageSnapshotPlugin(on, config);
};

// cypress/support/commands.js

import { addMatchImageSnapshotCommand } from 'cypress-image-snapshot/command';addMatchImageSnapshotCommand({
  failureThreshold: 0.00,
  failureThresholdType: 'percent',
  customDiffConfig: { threshold: 0.0 },
  capture: 'viewport',
});Cypress.Commands.add("setResolution", (size) => {
  if (Cypress._.isArray(size)) {
     cy.viewport(size[0], size[1]);
   } else {
    cy.viewport(size);
  }
 })

Writing & running the tests

The tests in this example are going to cover a basic responsive website of two pages, about & skills.

Create a new test spec in the cypress/intergration/ directory, ‘visual-tests.spec.js’.

describe('Visual regression tests', () => {
  it('Should match previous screenshot "about Page"', () => {
    cy.visit('/#about');
    cy.matchImageSnapshot();
  });
});

Open the cypres test runner with the npm script:

npm run cy:open

…click on the ‘visual-test.spec.js’ test to execute the spec. The test will take a snapshot of the about page which resides in the /snapshot/{spec-name} directory.

Now modify the webpage to create a visual difference and re-run the test. This will provide an example of what occurs during a test failure, revert the website changes after the test has run.

The failure result will appear in /snapshot/{spec-name}/__diff_output__ /{test_name.png} directory

Adding coverage of the second page

describe('Visual regression tests', () => {
  it('Should match previous screenshot "about Page"', () => {
    cy.visit('/#about');
    cy.matchImageSnapshot();
  });
  it('Should match previous screenshot "skills Page"', () => {
    cy.visit('/#skills');
    cy.matchImageSnapshot();
  });
});

Each ‘it’ block is its own test, appending .only to an ‘it’ singles it out for execution.

Striving to keep the code DRY, the following refactor can take place, this allows for the tests to be dynamically created, based on the number of pages fed into the ‘pages’ array.

const pages = [
  'about',
  'skills',
];describe('Visual regression tests', () => {
  pages.forEach((page) => {
    it(`Should match previous screenshot '${page} Page'`, () => {
      cy.visit(`/#${page}`);
      cy.matchImageSnapshot();
    });
  });
});

Adding responsive coverage

With dynamic test creation in mind, create a ‘sizes’ array to contain the desired viewports to be tested, Cypress has some predefined device sizes which can be used both portrait and landscape, alternatively, custom resolutions can also be provided.

In the below example we are looping through each size for each page, creating a snapshot with string interpolation used to name the test.

const sizes = [
  ['iphone-6', 'landscape'],
  'iphone-6',
  'ipad-2',
  ['ipad-2', 'landscape'],
  [1920, 1080],
];const pages = [
  'about',
  'skills',
];describe('Visual regression tests', () => {
  sizes.forEach((size) => {
    pages.forEach((page) => {
      it(`Should match previous screenshot '${page} Page' When '${size}' resolution`, () => {
        cy.setResolution(size);
        cy.visit(`/#${page}`);
        cy.matchImageSnapshot();
      });
    });
  });
});

For more advanced web apps…

…Which contain date pickers and constantly updating elements, e.g. a social feed, it is possible to take control of the date in the DOM and ‘freeze’ time with the cy.clock() Cypress function. It’s even possible to scrub out entire sections of the page with the blackout object passed into the screenshot function.

Below is an example of both the date being controlled and sections of the DOM scrubbed out.

const sizes = [
  ['iphone-6', 'landscape'],
  'iphone-6',
  'ipad-2',
  ['ipad-2', 'landscape'],
  [1920, 1080],
];const pages = [
  'about',
  'skills',
];const scrubbedElements = [
  '.lead',
  '.list-inline',
  '.img-fluid',
];describe('Visual regression tests', () => {
  sizes.forEach((size) => {
    pages.forEach((page) => {
      it(`Should match previous screenshot '${page} Page' When '${size}' resolution`, () => {
        const now = new Date(Date.UTC(2019, 1, 1)).getTime();
        cy.clock(now);
        cy.setResolution(size);
        cy.visit(`/#${page}`);
        cy.matchImageSnapshot({ blackout: scrubbedElements });
      });
    });
  });
});

Running the tests on the CLI

All of the above has been shown with the Cypress test runner locally.

You will want to run the tests on the command line so they can be plumbed into a Continuous integration workflow.

To run all of you Cypress tests you can execute

npx cypress run

you can also specify a single spec file, comma delimited lists or glob patterns

npx cypress run --spec "cypress\integration\visual-tests.spec.js"
npx cypress run --spec "cypress\integration\test1.spec.js","cypress\integration\test2.spec.js"
npx cypress run --spec "cypress\integration\**\*.spec.js

CLI test output

Updating the images

There will come a time when your tests discover a difference / ‘failure’ which is expected, the page has changed and it was desired.

This is when the benchmarked images will need to be updated. you can pass a parameter to overwrite all saved benchmark images with the following

npx cypress run --env updateSnapshots=true

Or pass in specific specs

npx cypress run --env updateSnapshots=true --spec "cypress\integration\visual-tests.spec.js"

That’s it!

With the above pointers, you could be well on the way to visually regression testing your web app!

You are welcome to pull down my code from the Norwich Node User Group GitHub page for a closer look.

Thanks for reading my experience 👀