Integrating BackstopJS in our Drupal updates workflow — part 1

As a support developer at LimoenGroen I work on a lot of websites. We like to keep all our websites safe and secure, so we perform Drupal updates on a monthly basis. We also want to make sure that the updates do not cause unintended changes. One of the tools we use to test that automatically is BackstopJS. We value Open Source at LimoenGroen and therefore we like to share our BackstopJS workflow. I will show you:

1. how easy it is to start using BackstopJS (this post)
2. what you can do to integrate BackstopJS in the CI of your choice (follow-up post)
3. how to perform tests as an authenticated Drupal user (follow-up post)
4. options and ideas to keep your tests organised and easy to maintain (follow-up post)

What is BackstopJS?

BackstopJS is the implementation of a very simple idea. Make a screenshot of your site site before and after you apply changes. Compare the screenshots and it reveals any difference. This is in a nutshell what BackstopJS does. You tell it to load a website, create screenshots, compare those screenshots with previously approved screenshots and report the results to you. And the best part? It is very easy to get it to work!

BackstopJS was created to “catch the CSS curveballs”, the unintended changes when you work on your CSS. That is why this form of testing is called visual regression testing. There are more tools (e.g. Wraith, Gemini or Spectre) out there that perform this. We have chosen BackstopJS for it is easy to install, it has a lot of configuration options, and it fits nicely in our workflow.

Getting started

BackstopJS is a npm package and as such you need to have both node.js and npm installed. It is also recommended to have the latest version of Chrome. If you’re all set, installing BackstopJS is very easy. If you’re new to BackstopJS it is recommended to install it globally:

npm install -g backstopjs 

Version: this article was written with BackstopJS v3.9.13

To start with your first BackstopJS tests, create a new folder and execute the following command in that folder:

backstop init

This will set up a basic structure with all elements in place to run a first test. The first thing you need now is the initial set of images that BacksopJS wil use to compare with, the reference images:

backstop reference

BackstopJS will report on the command line that new reference files will be created and will also tell you that you can now run a test:

backstop test

This time a browser will open and show you a report.

BackstopJS report for the default test configuration.

Congratulations, you’ve just finished the basic installation and performed your first test run.

BackstopJS basics

To get started with implementing your own tests or scenarios there are a couple of configuration options that you need to know. When you create a new BackstopJS project a file is created called backstop.json. This file contains the default configuration to run the sample scenario. I will briefly highlight the required elements in the backstop.json file and give you the minimum that you need to run a scenario.

Viewports

BackstopJS runs a headless browser to create the screenshots. You can tell BackstopJS what the default viewport sizes should be. If your test scenario does not have a viewport definition this default viewport definition will be used. The default backstop.json viewports looks like:

"viewports": [
    {
      "label": "phone",
      "width": 320,
      "height": 480
    },
    {
      "label": "tablet",
      "width": 1024,
      "height": 768
    }
  ],

All scenarios that do not have a viewport definition will be tested in two viewports.

Scenarios

A scenario is your test case. The default backstop.json file has a lot of configuration options, but you can leave out a lot of them. What each scenario needs is:

"scenarios": [
    {
      "label": "BackstopJS Homepage",
      "url": "https://garris.github.io/BackstopJS/",
    }
  ],

Something to note here is that anywhere you see “label” in your backstop.json file you should take care that it is unique. The label is used to generate a filename for the screenshot. When you have two scenarios with the same label, you will get unexpected results.

BackstopJS is also able to run partial tests based on a filter passed in on the command line. This filter is treated as a regular expression and applies to the labels of the scenarios. Setting up a logical naming convention for your filters will save you time later on.

In this article I will expand on the scenario configuration options, but if you want to know more take a look at the documentation.

Paths

The paths section in the backstop.json file is where you can tell BackstopJS where to store your reference and test images, where to save the reports and where to find the engine scripts for your scenarios. When starting out the basic configuration works, but you will probably want to change this later on. For example, when running a Drupal multisite installation you could set up the reference and test paths to reflect which website you are testing.

Minimum configuration

Putting it all together and to have a starting point for your own scenarios, here is the minimum configuration file that you need:

{
  "id": "backstop_default",
  "viewports": [
    {
      "label": "tablet",
      "width": 1024,
      "height": 768
    }
  ],
  "onBeforeScript": "puppet/onBefore.js",
  "onReadyScript": "puppet/onReady.js",
  "scenarios": [
    {
      "label": "BackstopJS Homepage",
      "url": "https://garris.github.io/BackstopJS/"
    }
  ],
  "paths": {
    "bitmaps_reference": "backstop_data/bitmaps_reference",
    "bitmaps_test": "backstop_data/bitmaps_test",
    "engine_scripts": "backstop_data/engine_scripts",
    "html_report": "backstop_data/html_report",
    "ci_report": "backstop_data/ci_report"
  },
  "report": ["browser"],
  "engine": "puppeteer"
}

This scenario will create a screenshot of the entire page (https://garris.github.io/BackstopJS/) using a tablet with viewport dimensions of 1024px wide and 768px high. Please note that the whole document will be in the screenshot.

Advanced configuration options

The option to compare screenshots of the whole document is the basic idea of visual regression testing. There are scenarios where you want to focus on certain elements of a page, or where some dynamic content changes on each page view. You can easily account for that with BackstopJS.

Elements and viewports

In almost all our projects we have elements that are only visible on a certain viewport size or have different behaviours depending on the viewport size. The main menu is a good example. On devices with a small viewport the main menu is usually hidden until you click on a link or “hamburger” icon. That link is not available on wider viewports, so it is unnecessary to have a scenario for it on all viewports.

To demonstrate this, add the following scenario to your backstop.json file:

{
  "label": "LimoenGroen Mobile Menu",
  "url": "https://www.limoengroen.nl",
  "clickSelector": ".overflow-toggler",
  "selectors": [
    "viewport",
    ".overflow-container"
  ],
  "viewports": [
    {
      "label": "mobile_only",
      "width": 320,
      "height": 480
    }
  ]
}

After running your reference and test commands, there will be two new screenshots in the report.

Report after adding the scenario to your backstop.json file

The scenario tells BackstopJS to load the LimoenGroen homepage in a mobile viewport, click the element with CSS class “.overflow-toggler” (clickSelector) and then create a screenshot of the viewport and a screenshot of the element with the CSS class “.overflow-container”. There will be no screenshots for this scenario for your default viewports.

The “selectors” option allows you to define which elements on the page you want to capture a screenshot for. There are two special values, document and viewport. The first is the default value and will capture the whole document. The latter will capture a screenshot of the viewport.

Advertisements and banners

When a website is using elements that are dynamic or vary per page view it can lead to false positives. Fortunately BackstopJS has the option to hide elements on or remove elements from a webpage. With the “hideSelectors” and “removeSelectors” option you can specify arrays of elements to hide or remove prior to capturing a screenshot.

Bypassing a cookiewall

One of our projects used a cookiewall. Access to the site was only allowed to visitors that clicked on the button. A cookie would then be set allowing the visitor to view the website on subsequent visits. Running the sample configuration on this website would have shown the same result for each scenario, a screenshot of the cookiewall page.

BackstopJS has a mechanism to pass on cookies to the website that is being tested. When you are using the vanilla configuration there will be a file called cookies.json in the backstop_data/engine_scripts subfolder.

[
  {
    "domain": ".www.yourdomain.com",
    "path": "/",
    "name": "yourCookieName",
    "value": "yourCookieValue",
    "expirationDate": 1798790400,
    "hostOnly": false,
    "httpOnly": false,
    "secure": false,
    "session": false,
    "sameSite": "no_restriction"
  }
]

You can replace the values in this file with the appropriate values for your website. One thing to note is that both the chromy and the puppeteer loadCookies.js script assume your website is using a certificate and the generated cookies will use https.

This covers the basic BackstopJS configuration options. In the next article I will dive into making the configuration more dynamic, enabling us to use BackstopJS in our CI setup.