Non visual regression in GitHub pull requests
At Talend, we maintain a Bootstrap theme for many of our applications. In order to quickly identify CSS regressions, we generate screenshots of each part of our HTML example. The goal is to be able to compare them with the reference for each pull request.
Github
Our Github workflow is quite simple: nobody can push on the master branch until, at least, one person approves. For that reason, it makes sense to help reviewers see the impact of the changes. We are lucky because GitHub offers three types of images comparison modes (2-up, Swipe and Onion Skin) so we can review any possible regression.
SlimerJS
SlimerJS allows you to interact with a web page through an external JS script.
And furthermore, it can take screenshots. It is like PhantomJS which is a headless WebKit scriptable using a JavaScript API. We chose SlimerJS over PhantomJS because we experienced some fonts and CSS transformations issues.
SlimerJS is similar to PhantomJs, except that it runs on top of Gecko, the browser engine of Mozilla Firefox, instead of Webkit, and is not yet truly headless.
So first of all, we installed it with:
yarn add --dev slimerjs
Then, we created test/screenshots.slimerjs.js
Finally, we added screenshots.config.json in order to store CSS selectors mapping.
Travis CI
Let Travis do the trick for all the pull requests.
After generating a Github API token and adding it into our Travis configuration, we create .travis/after_success.sh
How it works is quite simple, we start a static server and run SlimerJS test. If newly generated screenshots have changed (like git diff), Travis will commit them to the current git branch.
Finally we created .travis.yml to enable Travis CI
Conclusion
Now, we let SlimerJS identify visual regressions highlighted by Github committed by Travis for each pull request.
And for each of them, in addition of the quick preview of all impacts of the changes, we can be able to detect side effects. That definitely is a time saver!