Visual Regression Testing with Cypress.io + cypress-visual-regression plugin
In this post will share visual regression testing with cypress plugin i.e cypress-visual-regression. For one of the projects, we had a requirement to validate the drupal content management website images across browsers and also validate the dashboards and reports. For each build / or for any change, manually it’s hard to test for the entire site. We use cypress as our primary core framework to automate all test suites and we found visual regression for cypress, very useful to validate the dashboards/reports/images etc.
What is visual testing
How does it work
What problem does it solve?
Cypress automation tool framework
Install — cypress visual regression plugin
Run base images / actual and then compare.
What visual testing
Detect and view visual changes
Manual comparison
Visual bugs are hard to find
Colour variations
How does it work
Base image
Current and base image comparison
Variance / tolerance
What problem does it solve
Visual change control
Test and validate user interface
Images, Dashboards, Graphs
cypress-visual-regression plugin to solve the above problem
a. Plugin for cypress
b. Capture base line images
c. Run the script and compare actual vs base
d. Reporting
In previous post we have seen how to install Cypress (Please refer — https://docs.cypress.io/guides/getting-started/installing-cypress.html#yarn-add).
Once installed lets create simple framework that would launch https://www.phptravels.net/admin and login with valid credentials and create a CMS Pages record.
Go to the project folder where its created in your machine, under Cypress folder we can see below folder structure that holds fixtures, integration, plugins and support.
Let’s go ahead and create a simple framework (this may not be necessay for the visual regression still through of sharing this. Under fixture folder create “.js” files like credentials.js, data.js and selector.json
Create cypress.env file that has user credentials data.
Create a common methods in command.js file
Under integration folders let’s create login.js that would import selectors.json and credentials.js file and also it would call the method login() that’s created in command.js and also the login.js has a afterEach method that would logout from the portal.
Add cy.compareSnapshot('login'); in your tests specs whenever you want to test for visual regressions, making sure to replace login with a relevant name. You can also add an optional error threshold: Value can range from 0.00 (no difference) to 1.00 (every pixel is different). So, if you enter an error threshold of 0.51, the test would fail only if > 51% of pixels are different.
Create another test cases that would create the CMS Pages for https://www.phptravels.net/admin
Next let’s configure cypress-visual -regression plugin, basically cypress doesn’t provide builtin visual testing but using cypress-visual-regression that use PixelMatch library for image comparisons. This will capture the base image and then compare with current images for each run. So if any images difference then an error is thrown and will create “diff” folder where we can analyze the failures.
After installation we need to configure some of the files, please refer below. Basically we need to edit the files like cypress.json, index.js and command.js
Set base line images
Run comparison
After run a “diff” folder will be created under snapshots and that has the errors images for the portal.
Visual comparison for Login and CMS pages result would look like below..
As we can see there is an issue with cmspages.js use cases…
Conclusion
We have seen how we can leverage cypress-visual-regression free plugin to compare images for each run, but still it is challenge to integrate this with any integration tools like jenkins/ Circle CI etc.
Percy.io is the tool that would solve this problem, In next blog will share my experience working with Visual regression with Percy+CircleCI end to end integration.
References
Thanks
Gururaj
