Getting started with visual testing in CodeceptJS

Why do we need visual testing?

Functional testing can miss glaring visual bugs related to different elements, because they are rendering correctly in DOM and function as expected. However, these visual bugs can impact the brand and user experience because they are encountered by users before we can fix them. Generally, visual tests need to be verified manually, but we can save time by automating it by applying visual testing.

Another reason that may cause bugs is that not every browser engine supports CSS the same way. They can render the CSS differently and only offer one way to review your application with different resolutions and on different browsers — by manually opening your application and checking yourself. Visual testing is ideal solution to automate this process.

Such a bug would be immediately found during manual testing, but automated functional testing would miss it because the bug is merely visual and does not affect the button’s performance. Usually, visual bugs never break an application, but they are easily found by users. We can’t say for certain that our application works properly without visual testing, especially when the frontend is going through constant changes. Such bugs will only become apparent after deploying any changes to our prod environment. Visual testing can help us to see how our users experience our products. Moreover, screenshot testing provides an opportunity for testing on different devices, browser resolutions, orientations.

How do visual testing tools work?

When running tests in browsers, visual testing will create screenshots that we can use to verify all elements look and work as expected. Such way we’re getting a current image and we need to have an etalon, base image. Generally, we set the base image with the visual testing tool, and it is commonly located in the project files. Thus, we have a base and current image, and our goal is to compare them to see the difference between them based on a predefined tolerance level. Basically, this is how every visual testing tool works.

Visual tests can be flaky and unstable due to a number of reasons. One of them is — different operating systems use different fonts. Below, you can find a list of issues with visual testing that we have to solve.

Issues with visual testing:

• scrolling
• working with dynamic content
• video content testing
• working with animations
• handle cursor blinking
• how to handle dates
• browsers render web applications differently depending on OS

To avoid these issues, we should:

• utilize Mock API if we have dynamic data and date, in order to get the same data consistently
• disable cursor, CSS animation, and scrolling for visual tests
• use tolerance level to handle strict pixel to pixel comparisons
• apply component comparisons
• Screenshots are different on different OS’s, therefore, we need to use the same environment to perform visual testing or use docker images with certain browser versions to get the same result locally and in CI

How should we write visual tests?

W e can usually incorporate visual testing into E2E test suites by simply expanding our scenarios with screenshots for important pages or regions, depending on our goals. We should test our user scenarios as usual, and the visual test’s framework will take a screenshot to use for comparison between the base and new image.

There are a number of tools, which help with visual tests. Let’s consider the ones that are available for CodeceptJS.

Applitools

Applitools is an AI-powered visual testing and monitoring platform which automatically validates the look and user experience of your apps and sites.

Pros:

• Stable testing — you can run thousands of tests and they will remain stable
• Smart algorithms based on artificial intelligence utilizing machine learning (ML) and deep learning (DL) models — Applitools features a network of hundreds of algorithms that uses different tools and approaches based on deep learning.
• Simple integration with automation tools (Selenium, Appium, CodeceptJS, WebdriverIO, and Cypress) for different languages and with various continuous integration systems (Bamboo, Jenkins, Travis, and others)
• Helps to avoid writing a lot of code by making screenshots
• Simple to write tests

Cons:

• Price (pretty expensive)
• Reports small deviations from base images, especially for dynamic content, as false failures

Visual Regression Tracker

Visual regression tracker — is an open-source, self-hosted solution for visual testing and managing results of visual testing.

Under the cover, the server is hosting a pixelmatch library. This library takes a base64 string and sends it to the server, which converts the string to a png file and compares it to the baseline picture. It then conducts a pixel to pixel image comparison and calculates what percentage of pixels do not correspond to the result. If this percentage exceeds the predefined tolerance limit, the test will fail because the result will surpass the accepted limit of discrepancies.

If we work with dynamic content, our tests will fail each time. We can avoid this by adding an option to ignore some of the regions in a screenshot.

We can also upload Visual Regression Tracker to our internal networks to avoid security issues with sending your images to external servers.

Pros:

• Independent from automation frameworks — no need for integration or to stick with a specific automation tool
• Independent from platforms — available on web, mobile, and desktop
• Baseline history — track the changes in baseline images
• Ignore regions —improves stability by ignoring unimportant or uncontrollable parts of the image
• Language support — JS, Java, or any other via REST API
• Easy setup — everything is inside Docker images
• Self-hosted — Keep your data saved on your network
• Support branches
• Open-source and free solution

Cons:

• Poor UI
• Algorithm isn’t as smart as Applitools

ResembleJS

Resemble.js is a Javascript tool compatible with CodeceptJS, which is used to compare and analyze images. Resemble.js analyzes and compares images with the help of HTML5 canvas and JavaScript.

By setting the mismatch tolerance level, we are also setting the valued to be used when comparing baseline images and screenshots. All future tests will take this comparison into account. Resemble.js uses a HTML5 File API to parse image data, and canvas for rendering image diffs. ResembleJS’s helper supports real browsers, headless browsers, and cloud browsers.

Pros:

• image comparison and basic analysis
• can limit regions to reduce the area of comparison

Cons:

• no web interface
• only supports pixel to pixel image comparison, which can lead to flaky tests
• based on PhantomJS, which is not supported anymore

Suggestions

• Avoid relying solely on visual tests. Visual testing help extend our test coverage, but we should continue conducting UI E2E tests as well.
• Perform screenshot testing for components, but avoid applying it for too small parts of pages
• Ignore dynamic content, disable cursors, CSS animations, and scrolling with visual tests
• Adjust the tolerance level to correspond to human expectations because strict pixel to pixel comparisons will lead to flaky tests

The next article will feature a practical guide about how we can integrate these tools to CodeceptJS and how to conduct visual tests with it.