Automated visual testing — how to improve UI verification
How many times have you or your teammate noticed wrongly placed button with different colour, and font not looking as previously expected right after deployment on production?
In order to prevent ourselves from such scenarios, we can use various types of tests focusing on graphical interface.
It is very popular among testers to use functional test for UI test in order to verify end-to-end behaviour of the system. Unfortunately, very often it appears to be forgotten that this kind of test does not fully test User Interface, but only checks a part of it. Nevertheless, visual tests are becoming more common in software development from year to year.
What is visual regression?
Visual regression mostly focuses on visual aspects of User Interface (UI). Verification of visuals can be made manually or with a help of automation. And regardless of the way you choose, they aim is to ensure that graphical interface has no flaws. It is crucial to review application also from that point of view. It also seems very important not to mix regression tests with visual regression. It does not equal.
In general, regression tests are designed to verify whether any change introduced into developed system does not break it. Although, we can say that visual regression can be a part of regression suite.
At present, almost all programming languages used for writing automated test are able to support visual tests. There are many tools and libraries that help testers and developers in building tests scenarios, but all are focused on:
- Properly rendered and displayed graphics on the page.
- Components of a page displayed correctly, e.g. buttons with proper colour and proper shape.
- Layout of a page e.g. each component is in right place on a page.
- Validation of component, if it does not overlay each other.
On the other hand, we take following aspects of application into consideration in functional tests:
- Validation of input fields.
- Page navigation check.
- Business logic check, e.g. checking CRUD operation on a page.
- Check if hyperlinks are working.
Considering both, functional tests and visual tests, they provide development team with strong tool verifying changes during software development life cycle.
How it works?
The idea of visual regression is very simple. First of all, it is important to create baseline — it is first screen shot capture of each step. Secondly, run another round of test to check whether implemented changes do not break application UI. When test run is finished, we need to compare latest test run with baseline, so to verify appearance of any changes. If so, then we need to check whether appearing difference is something what development team intendent to do. In other words. we need to check if changes appeared for the reason. If it turns out that development team did not intend to implement any of them, then they are called visual bugs.
In the last phase, baseline should be updated with new changes in order to validate new regression iterations properly.
How to use it?
There are few tools on the current market which support visual testing. The most popular are:
Let’s consider having tests written in Java. For image comparison and image validation we used Applitools Eyes tool.
Applitools Eyes provides various utilities which perform visual tests. Not only proper library for taking and comparing taken pictures of a page can be found there, but also fancy dashboard which helps you with managing your tests. The dashboard allows you to mark the test as passed or as failed which automatically leads to ticket creation in your issue tracker.
All you need to initiate image comparison is turning Applitools on in your test code.
Let’s assume that we have simple test class with one test that opens page and goes to one predefined tab on that page:
As you can see, test appears to be very simple. It opens Firefox browser, navigates to www.siili.com page and clicks on Stories tab. After that, browser closes and test is finished.
To add visual verification, we need to initiate Applitools object in our code and then invoke method allowing us to capture picture. In code it looks like the one below:
In order to use Applitools Eye, you need to open connection with Applitools cloud server and pass few parameters to create proper baseline.
In method:
eyes.open();you can convey 5 arguments which will create baseline. These arguments are:
- Browser name,
- Operating system,
- Application name,
- Test case name,
- Viewport.
As mentioned in the above code, there are only 3 arguments passed in the method. Browser name and operating system can be obtained from driver instance. In the above, viewport was omitted and system took default one.
To take screenshot of each step following method is being invoked:
eyes.checkWindow();in which a string is an argument which defines test step. Each check has default timeout (2 seconds) which can be changed by passing proper value as method argument, e.g. checkWindow(“ToDoStep”, 1000);
in above example, wait is set to 1 second.
Now when test has been executed, we can go to Applitools dashboard and check the results.
In www.applitools.com, when you log in, you will be automatically redirected to your latest dashboard.
List of test cases or test suite that has been lately executed is displayed in the left pane (1). In center view user can observe test cases within suite and its statuses.
In test result view, it is possible to manage each test step and verify whether it has any flaws or not. Each step can be viewed with baseline comparison (1) or without it. When Applitools detects occurrence of any differences comparing to baseline, each picture is marked with pink marker and changes are selected. It is user’s decision whether change is desired, whether not. If user decides on the change to be correct, he marks it with thumb up icon (2). When change appears to be a visual bug, then user marks test step with thumb down icon (3). In that scenario, the test step is marked as failed and entire test case is marked as failed. When step is verified it can be saved (4) and user can go to another test step in order to verify it.
Where to use visual tests?
Visual tests can be used in any kind of applications. Nevertheless, we need to keep in mind that every test automation requires time to prepare proper code test cases, especially automated tests, and even visual ones.
It is highly recommended to use visual test in application in which customer puts strong impact on graphical aspects. It is very useful to use visual tests in CMS applications where content may change very often, or it depends on the country it is published in.
It is highly impossible or at least time consuming to verify application available for few countries with different languages. This is especially likely when trying to check behaviour of application on supported browsers. Moreover, it is highly recommended to use visual tests in those applications which support both web and mobile view (RWD). Checking manually all web browsers and all mobile devices, making sure that they are deprived of any visual flaws, appears to be very hard. Here automatic visual regression tests come with help.
