Adding visual regression testing to Sidekiq with Percy.io
Sidekiq has taken the background job processing world by storm. Sidekiq’s creator, Mike Perham, has a near-obsessive focus on performance, and it shows in how quickly Sidekiq can crunch through any jobs you throw at it. It’s very popular in the Ruby ecosystem. Recent reports show Sidekiq is now more widely used than Delayed Job and Resque, the earlier generation background processing systems.
This article briefly describes what visual regression testing is, and how Percy was added to Sidekiq’s test suite to run visual regression tests, and notify of visual changes in GitHub pull requests.
Sidekiq’s web interface
First, let’s introduce Sidekiq’s web UI. While Sidekiq is working away processing jobs, it’s reassuring to know what’s going on. The web UI shows you how many jobs Sidekiq has processed, how many jobs it has queued, how many workers are busy etc.
What is visual regression testing and why add it?
Visual regression testing ensures that your software looks how you intend it to look. It involves taking snapshots of your UI, making changes to your code via the work you do, then taking another set of UI snapshots, and comparing them to the previous snapshots taking to determine what changed visually.
As an example, if we were doing a CSS refactor and we accidentally made a CSS change to Sidekiq’s UI to remove the white background from the summary bar, Percy’s visual review system would highlight the part of the page that has changed:
Clicking on the highlights reveal the underlying screenshot to show how the summary bar changed (it lost the white background):
When making changes to Sidekiq’s web UI, or any UI, it’s invaluable to know that the UI is being presented as you expect, and that you’re not shipping visual bugs to customers.
Refactoring code, adjusting CSS rules, or altering a completely different part of the app can sometimes introduce unexpected and unwanted visual side effects.
Adding visual regression tests results in confidence when making changes. You can see that what you intended to change has changed (ie. the green button got bigger) and there are no unexpected changes (ie. the other buttons stayed the same).
What is Percy?
Percy is a visual review platform. It makes it easy to add visual regression tests to your application and notifies you when visual changes have occurred. Percy provides a platform to review and approve visual changes, and integrates right into your GitHub pull requests.
Adding Percy to Sidekiq
To add Percy to Sidekiq, we made some small adjustments to it’s testing system. This section explains why.
Sidekiq’s web interface is built as a Rack application. It has a suite of MiniTest tests, and uses Rack::Test to test the web app.
Rack::Test is very fast, and is great if you only care about the html in the response.
However, for visual testing we need more — we need a real browser to be running in tests, handling DOM updates and running JavaScript. We need the page state to be full and complete, but Rack::Test stops short of running a browser or JavaScript. It simply interacts with Rack handlers and HTML directly.
To solve this and get an accurate DOM snapshot, we switched Sidekiq’s UI tests to use Capybara with the PhantomJS headless browser. The Poltergeist driver for Capybara makes working with PhantomJS painless. This allows the page to fully render with javascript updates to the DOM, before taking the snapshots with Percy.
Introducing the headless browser is asking the tests to do more work, so the test suite takes a little longer to complete. Sidekiq’s 279 tests took ~1 second to run with Rack::Test, and they now take ~4 seconds with Capybara and PhantomJS.
To switch to Capybara, in addition to adding the Capybara and Poltergeist gems, we had to make some modifications to the test code.
Switching from Rack::Test to Capybara and Percy
Before updating, the tests using Rack::Test looked like this:
After switching the tests to use Capybara, and adding Percy to take a snapshot, the tests now look like this:
We also adjusted Sidekiq’s test helper to initialize and finalize Percy:
Visual reviews in GitHub pull requests
Now when Sidekiq’s test suite runs in its CI environment, MiniTest, Capybara, and Percy co-operate to take DOM snapshots. The DOM snapshots, supporting images, and CSS resources are automatically uploaded to Percy. Percy immediately gets to work rendering the DOM snaphots in a custom browser farm, capturing screenshots, and running comparisons on them with screenshots from an earlier build.
If Sidekiq has an open pull request associated with the current git commit, Percy updates the pull request status with the results of the visual regression tests, showing whether there are any changes that require review.
When there are changes, Sidekiq developers can click the ‘Details’ link to open Percy and review the changes.
Once the Sidekiq developers are happy Percy’s only showing intended changes, the build can be approved in Percy and the status shown in the pull request will update showing the approval.
Team Sidekiq can now merge that pull request and ship with confidence that Sidekiq’s UI will display as intended.
If you enjoyed this article, please share it on Twitter or email it to a friend.
If you’d like to use Percy to add automated visual review and visual regression testing to your own app, we’d be happy to help. Chat with us on Percy.io, or email us at hello@percy.io.
Links to the technologies mentioned in this article:
- Sidekiq
- Percy.io
- Capybara, Poltergeist, and PhantomJS
- Rack and Rack::Test
- MiniTest
- Travis CI — the CI service Sidekiq uses. Travis CI also uses Percy.
