Automated Visual Testing With Snapshots: Part 2
The beautiful part of the structure we built actually starts here. Let’s say a new feature has been developed and a new button has been added to some page and your snapshot tests detect it and fail.
You have to update the tests in traditional tests, right? It’s as easy as clicking the button here. Since the reference image will be updated, you are updating the tests without writing any code.
Customize Snapshot Testing Library
We forked the swift-snapshot-testing library and customized the path to the file folder where we will save the reference and failure files. We changed the reference and failure image paths for Jenkins. We didn’t store images inside the project to avoid an increase in project size. In this way, we can store Snapshot images in the same virtual machine in a folder outside the project.
We created a new lane for snapshots in FastFile and separated the test plan.
Jenkins Script
We trigger the tests thanks to a job we created in Jenkins. The script working in this job is as follows:
Before running tests, we clean the folder where we keep the snapshot images that fail. Images of failed tests will be added back to this folder. After running all the tests, we need to check if there are any files in this folder. For this, we get the names of the files with a small terminal code(line 11). If there is any filename, it means the test is failed.
Since there are processes running on the virtual machine, we let two simulators run in parallel to ensure stability.We couldn’t access the error file from outside of the virtual machine. To upload the image of the error file to Slack as a message, we send a request in the Jenkins script.(line 16)
We added 3 buttons in Slack message.
- In order to see the reference file we added Show Reference Picture button.
- To able to update the reference file of a Snapshot test, we display Update button using the Slack API in the service we developed and send them to the Slack channel.
- For reasons similar to the rerun feature that came to XCode 13, we added a Rerun Test button for a single test to run again. Using the feature that build-for-testing provides us, we can control the test without re-build by specifying a single test name.
The resulting image showing the difference of failed and reference files may not be very explanatory. That’s why we decided to send the reference and failure images to the channel separately. In cases where we couldn’t understand the difference, we can look at the difference between the two images manually with the images comparison tools.
Update Reference Image
To update the reference images, we trigger the Jenkins job below. If the job finishes with success, we send a success message to Slack via the service we use.
At the beginning of this process, we followed the results of all tests with the QA team on a common slack channel. We made a POC first then ran all the tests and iterate every day.
We realized that this process wasn’t practical and requires a lot of effort so we divided the responsibilities by channel and increased the traceability.
We prepared these tests separately for 5 different channels in Trendyol and created a separate Slack channel. If all tests in the channel run successfully, we send a notification for that result.
What is Next
- Sending difference images to slack channel using image exporter tool.
- Adding a new button to the Slack message of failed Snapshot tests. That will automatically open the bug task by adding the name and picture of the test to the Jira of the related team’s board.
In conclusion, we added Snapshot testing in CI/CD process. Thank you for reading. If you have any questions, feel free to contact me and Atakan Karslı.
