Building a simple pixel match test infrastructure for your needs
Image diff, visual testing, pixel match, or whatever you call it, believe it or not, will become a thing. There are times when you need to keep your web application consistent regardless of platform. Emergence of multi-platform front-end technologies such as React Native, Flutter, or maybe PWA will make it even worse.
So, what is a pixel match test?
So, what is pixel match testing?
Basically, a pixel match testing is a test where you compare two image and determine if there is a difference between the image. The image will be fetched from screenshot of your app, and then with image comparator library. It will do the work and compare those image for you.
The image is usually divided to two kinds of image. That is the baseline image and the build image. Build image is basically a set of image that will be gathered if you run the test. It’s basically a set of screenshot of your app in that specific build. And then, there is a baseline image that will be used as a base to your current build image. Similarity between build images and baseline images is your aim in this testing method.
If your build image differ very much from the baseline image. It could meant two things. First, is there is an error in your build that is make the image significantly different. Or maybe there is a major change that you did in this build (revamp, for example), So you have to update your baseline image according to that changes you made.
If you look around. There is a lot of people that actually offer this kind of testing for you. For example, company like percy.io and applitools.com offer this kind of service for you. But sadly it’s not free. So you whether want to pay for it, or build one for yourself.
How we do it
We made pixel match infrastructure according to this design.
In short, The architecture are following this steps.
- Run your UI automation test. Take screenshot on the page where you want to compare the UI.
- Send the image to Image Comparer Service, where it will be processed to diff image.
- Diff Image will be sent back to the Automation Test Runner.
- Generate test report in Automation Test Runner.
- Send Diff Image and test report to Telegram.
For image comparer service, We use Piksel. You can access it here if you want to take a look : https://github.com/luthfiswees/piksel
For Automation Test Runner, we use Jenkins that runs integration test using Cucumber and Capybara. For details, you can also look it up here
- Capybara : https://github.com/teamcapybara/capybara
- Cucumber for Ruby on Rails : https://github.com/cucumber/cucumber-rails
In case you are wondering about how we set the baseline, Piksel will automatically set your first image as a baseline image. Piksel use tags to differentiate between image. So, for every tag, there is only one baseline image.
For example, if you want to send image A with tags “XYZ” for the first time. Piksel will set image A as your baseline image. And if you send another image with tag “XYZ”, all of those image will be compared to image A.
So, it is possible to change the baseline later? You can do it by accessing the endpoint to change the baseline later. If you want to know more about it, you can see it in the README.
The result
Here are the difference between screenshot image and diff image.
The difference between the screenshot and the diff image is that the diff image will highlight the difference between screenshot image and baseline image that has been set. This highlight will help you in spotting difference between images.
For the test report we made it look like this.
This report will show you the diff image and the image tag. The image tag is used to differentiate between different screenshots.
Conclusion
I think to build a pixel match testing infrastructure is a good thing to do. It offers you a different insight than other testing methods like unit testing and standard integration testing. It gives you insight about how different the UI in each build and try to measure those difference. And to make it even better, it’s not that hard to build one.
Maybe, if you have a plan to build an app that will run on multiple platforms. Or maybe you just curious about this. Why don’t you give it a try? Maybe it will pay off on the long run!
