Umami distribution testing automation with and Diffy

Yuriy Gerasimov
Jun 19, 2019 · 3 min read

One of the big things that drive Drupal 8 adoption is an easy ability to “try it out”. That is a fantastic Umami distribution project.

Lately, we were working on automating visual regression testing so maintainers have an easy way to check the patches.

The only thing that is needed right now is to copy patch’s URL, paste it into one of the test branches in GitHub project — and in 5–10 minutes you’ll get a report with all visual changes that this patch brings. And even more — visual testing covers both anonymous users view and an admin as well.

GitHub set up

The main repository is available at .

As you could see it is fairly standard composer based Drupal setup.

Main logic resides in composer.json file. It uses latest Drupal 8.7 dev branch and whenever you like to test the patch there — you simply add it in “patches” section: set up has provided an account for testing purposes. Thanks to G we have configured “tracking” three branches: master and three testing branches.

Once any commit lands (i.e. testing a patch) related environment gets rebuild. All logic resides in file.

Here are the hooks that get triggered on each deployment

So you can see that we:

  • reinstall the site from scratch by using umami distribution
  • run cron
  • set up some debugging information
  • trigger visual regression testing

Visual regression testing with Diffy

does a few things:

  • gets all the credentials from variables
  • checks if the current branch is master or one of the test branches
  • if it is a master branch — simply create screenshots
  • if it is a test branch — create screenshots and compare them with latest screenshots from master branch environment (production in terms of Diffy)

Once jobs are completed we send an email with notification. Also, the diff gets renamed based on the branch name. So it is easy to track what environment was tested and when.

Image for post
Image for post
Diff report for test branch

As we are testing Umami with authenticated user — Diffy is configured to have two separate projects. The one that covers admin user tests layout builder URL only, but it can cover any other pages that need to be tested.

Image for post
Image for post
Admin authenticated user

Security and passing variables from to Diffy

There was one very valid argument raised by what about security? We do publish admin password in public GitHub repository.

The way we solved it is we have set up HTTP basic authentication in and passed credentials to our script that triggers jobs.

Image for post
Image for post allows passing variables to your build scripts

So then even with having GitHub repository open nobody without passwords would be able to access our builds and login.

Other use cases

Because this set up is general composer based site it is fairly easy to have it for other drupal projects / distributions. So you could test themes, modules or any other products that have some changes in UI that you’d like to track., visual regression testing platform

Deployments free of visual bugs

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store