Non visual regression in GitHub pull requests

At Talend, we maintain a Bootstrap theme for many of our applications. In order to quickly identify CSS regressions, we generate screenshots of each part of our HTML example. The goal is to be able to compare them with the reference for each pull request.

Github

Our Github workflow is quite simple: nobody can push on the master branch until, at least, one person approves. For that reason, it makes sense to help reviewers see the impact of the changes. We are lucky because GitHub offers three types of images comparison modes (2-up, Swipe and Onion Skin) so we can review any possible regression.

SlimerJS

SlimerJS allows you to interact with a web page through an external JS script.

And furthermore, it can take screenshots. It is like PhantomJS which is a headless WebKit scriptable using a JavaScript API. We chose SlimerJS over PhantomJS because we experienced some fonts and CSS transformations issues.

SlimerJS is similar to PhantomJs, except that it runs on top of Gecko, the browser engine of Mozilla Firefox, instead of Webkit, and is not yet truly headless.

So first of all, we installed it with:

yarn add --dev slimerjs

Then, we created test/screenshots.slimerjs.js

Finally, we added screenshots.config.json in order to store CSS selectors mapping.

Travis CI

Let Travis do the trick for all the pull requests.

After generating a Github API token and adding it into our Travis configuration, we create .travis/after_success.sh

How it works is quite simple, we start a static server and run SlimerJS test. If newly generated screenshots have changed (like git diff), Travis will commit them to the current git branch.

Finally we created .travis.yml to enable Travis CI

Conclusion

Now, we let SlimerJS identify visual regressions highlighted by Github committed by Travis for each pull request.

And for each of them, in addition of the quick preview of all impacts of the changes, we can be able to detect side effects. That definitely is a time saver!