Visual Testing: Argos & Playwright
Hello everyone!! In this article we will explore how we can setup visual testing in our continuous integration workflow on Github using Argos CI and Playwright integration. Let’s get started. Before that, I will give a short overview on Visual testing and Argos CI.
What is Visual Testing?
Visual Testing is a software testing technique that mainly checks for visual aspects of a software application’s user interface (UI). Visual testing aims to verify that the application’s visual components like colors, padding, images, image sizes, fonts etc is displayed correctly and consistently across a wide range of devices, operating systems, and browsers.
Intro to Argos-CI
Argos provides the developer tools to debug tests and detect visual regressions directly from our CI setup and it can be easily integrated with a wide range of testing frameworks using Playwright, WebdriverIO, Cypress, Puppeteer etc.
Below mentioned are some of key use cases taken from their docs:
One more benefit of using Argos is that in case of test failures we can easily view failure screenshots, playwright traces directly in the Argos app itself which helps us during debugging. Also, its 100% open source and has good community support. For more information on Argos, you can refer docs.
Setup Overview
So for demo purposes I will configure and use a sample Next-js demo application which will be running in my local machine on port 3000. Then I will go ahead and install playwright and then configure Argos CI in our project. Once setup is done on initial run Argos will capture a reference baseline image (snapshot) for the landing page. Then, we will go ahead and make some minor changes in our home page, so when the next build is run in Argos using the changes done, the baseline image snapshot will not match with the new changes and we won’t be able to our PR with the changes done since visual checks have not passed.
Step-1: Setup boilerplate Next-js application
To configure a sample Next.js application, ensure that you have node installed in your machine, then create a project folder in any location of your machine and then open up the same folder in Visual Studio Code. Open the Terminal and run below command.
npx create-next-app@latestI have specified the below configurations as seen below selecting the default options presented, then it will go ahead and install and the relevant dependencies and packages required for the sample website.
To start the website on local server, run the below command.
npm run devOn running the above command, it will take a few seconds to compile the project and our website will be up and running on localhost port 3000.
This is just a sample page website with few links and images available.
To make changes in website, edit the app/page.tsx file available
Step-2: Installing Playwright
Next, we will go ahead and configure playwright in our project folder, run the below command to setup and configure playwright.
npm init playwright@latest
Once the playwright setup is done, we can just verify if we are able to run the boilerplate tests available under tests folder using the below command.
npx playwright test
Step-3: Installing argos-ci & setting up playwright config file.
Next, we will install argos-ci/playwright package using command
npm install --save-dev @argos-ci/playwright Now, we will go ahead and make few updates in playwright.config.ts file Fistly, updating the reporter section to add argos-ci reporter.
reporter: process.env.CI ? [["@argos-ci/playwright/reporter"]] : 'html',Specifying base url and adding trace and screenshot capabilities.
use: {
/* Base URL to use in actions like `await page.goto('/')`. */
baseURL: 'http://127.0.0.1:3000',
trace: 'retain-on-failure',
screenshot: 'only-on-failure'
},Configuring webserver as we will be running tests on our local setup.
webServer: {
command: 'npm run dev',
url: 'http://127.0.0.1:3000',
reuseExistingServer: !process.env.CI,
},Then, update the example.spec.js file available under tests as below
import { argosScreenshot } from "@argos-ci/playwright"
import { expect, test } from "@playwright/test"
test('Verify home page designs', async ({ page }) => {
await page.goto("/")
await expect(page.getByAltText("Vercel Logo")).toBeVisible()
await expect(page.getByAltText("Next.js Logo")).toBeVisible()
await argosScreenshot(page, "Homepage")
})What we are doing in the above test is navigating to our local host base url which is running on port 3000 and verifying visibility of the image logos for Vercel and Next.js. Then we are just taking a snapshot of the page using argosScreenshot method passing in the page fixture and the snapshot name which is “homepage”. This will be considered as our baseline image (golden image) which will be used to further visual checks.
Next, we will create an remote repository in Github and push our project code to Github. Also we will need to import this repository in argos-ci. You can login to Argos using your Github account.
In the Dashboard page, click on the “Create a new Project” button and import the repository which you have created. This will then take you to the builds page and you will be able to see your ARGOS_TOKEN. Then, push your project code from local to remote repository which will trigger the github actions workflow yml file and run all the configured tests.
Also in argos-ci project dashboard, you will see that a build will be generated for the workflow run, since this is the first run this will be considered as the Reference build.
On opening up the Reference build #1 you will be able to view the captured image along with some other options like expand the screenshot, hide changes overlay, show baselines and changes side by side etc in the top right corner of the screen to check snapshot images.
Step-4: Updating website and checking visual checks in argos
Now, we will proceed to make some minor updates in the page.tsx file, let’s create a new branch using the below command.
git checkout -b page-updatesOpen up the page.tsx file and make few updates, firstly we will just update the height and width of the Vercel and Next.js logo image and add some extra text to line 8 and line 56, the complete update page.tsx is as below.
import Image from "next/image";
export default function Home() {
return (
<main className="flex min-h-screen flex-col items-center justify-between p-24">
<div className="z-10 max-w-5xl w-full items-center justify-between font-mono text-sm lg:flex">
<p className="fixed left-0 top-0 flex w-full justify-center border-b border-gray-300 bg-gradient-to-b from-zinc-200 pb-6 pt-8 backdrop-blur-2xl dark:border-neutral-800 dark:bg-zinc-800/30 dark:from-inherit lg:static lg:w-auto lg:rounded-xl lg:border lg:bg-gray-200 lg:p-4 lg:dark:bg-zinc-800/30">
Get started by editing the file
<code className="font-mono font-bold">app/page.tsx</code>
</p>
<div className="fixed bottom-0 left-0 flex h-48 w-full items-end justify-center bg-gradient-to-t from-white via-white dark:from-black dark:via-black lg:static lg:h-auto lg:w-auto lg:bg-none">
<a
className="pointer-events-none flex place-items-center gap-2 p-8 lg:pointer-events-auto lg:p-0"
href="https://vercel.com?utm_source=create-next-app&utm_medium=appdir-template&utm_campaign=create-next-app"
target="_blank"
rel="noopener noreferrer"
>
By{" "}
<Image
src="/vercel.svg"
alt="Vercel Logo"
className="dark:invert"
width={200}
height={34}
priority
/>
</a>
</div>
</div>
<div className="relative flex place-items-center before:absolute before:h-[300px] before:w-full sm:before:w-[480px] before:-translate-x-1/2 before:rounded-full before:bg-gradient-radial before:from-white before:to-transparent before:blur-2xl before:content-[''] after:absolute after:-z-20 after:h-[180px] after:w-full sm:after:w-[240px] after:translate-x-1/3 after:bg-gradient-conic after:from-sky-200 after:via-blue-200 after:blur-2xl after:content-[''] before:dark:bg-gradient-to-br before:dark:from-transparent before:dark:to-blue-700 before:dark:opacity-10 after:dark:from-sky-900 after:dark:via-[#0141ff] after:dark:opacity-40 before:lg:h-[360px] z-[-1]">
<Image
className="relative dark:drop-shadow-[0_0_0.3rem_#ffffff70] dark:invert"
src="/next.svg"
alt="Next.js Logo"
width={200}
height={87}
priority
/>
</div>
<div className="mb-32 grid text-center lg:max-w-5xl lg:w-full lg:mb-0 lg:grid-cols-4 lg:text-left">
<a
href="https://nextjs.org/docs?utm_source=create-next-app&utm_medium=appdir-template&utm_campaign=create-next-app"
className="group rounded-lg border border-transparent px-5 py-4 transition-colors hover:border-gray-300 hover:bg-gray-100 hover:dark:border-neutral-700 hover:dark:bg-neutral-800/30"
target="_blank"
rel="noopener noreferrer"
>
<h2 className={`mb-3 text-2xl font-semibold`}>
Docs{" "}
<span className="inline-block transition-transform group-hover:translate-x-1 motion-reduce:transform-none">
->
</span>
</h2>
<p className={`m-0 max-w-[30ch] text-sm opacity-50`}>
Find all the in-depth information about Next.js features and API.
</p>
</a>
<a
href="https://nextjs.org/learn?utm_source=create-next-app&utm_medium=appdir-template-tw&utm_campaign=create-next-app"
className="group rounded-lg border border-transparent px-5 py-4 transition-colors hover:border-gray-300 hover:bg-gray-100 hover:dark:border-neutral-700 hover:dark:bg-neutral-800/30"
target="_blank"
rel="noopener noreferrer"
>
<h2 className={`mb-3 text-2xl font-semibold`}>
Learn{" "}
<span className="inline-block transition-transform group-hover:translate-x-1 motion-reduce:transform-none">
->
</span>
</h2>
<p className={`m-0 max-w-[30ch] text-sm opacity-50`}>
Learn about Next.js in an interactive course with quizzes!
</p>
</a>
<a
href="https://vercel.com/templates?framework=next.js&utm_source=create-next-app&utm_medium=appdir-template&utm_campaign=create-next-app"
className="group rounded-lg border border-transparent px-5 py-4 transition-colors hover:border-gray-300 hover:bg-gray-100 hover:dark:border-neutral-700 hover:dark:bg-neutral-800/30"
target="_blank"
rel="noopener noreferrer"
>
<h2 className={`mb-3 text-2xl font-semibold`}>
Templates{" "}
<span className="inline-block transition-transform group-hover:translate-x-1 motion-reduce:transform-none">
->
</span>
</h2>
<p className={`m-0 max-w-[30ch] text-sm opacity-50`}>
Explore starter templates for Next.js.
</p>
</a>
<a
href="https://vercel.com/new?utm_source=create-next-app&utm_medium=appdir-template&utm_campaign=create-next-app"
className="group rounded-lg border border-transparent px-5 py-4 transition-colors hover:border-gray-300 hover:bg-gray-100 hover:dark:border-neutral-700 hover:dark:bg-neutral-800/30"
target="_blank"
rel="noopener noreferrer"
>
<h2 className={`mb-3 text-2xl font-semibold`}>
Deploy{" "}
<span className="inline-block transition-transform group-hover:translate-x-1 motion-reduce:transform-none">
->
</span>
</h2>
<p className={`m-0 max-w-[30ch] text-sm opacity-50 text-balance`}>
Instantly deploy your Next.js site to a shareable URL with Vercel.
</p>
</a>
</div>
</main>
);
}The changes done can be verified locally as seen in the snapshot below.
Now push the changes from local to remote repository and create a new pull request with these changes which will trigger automated checks for playwright and argos in the pull request worflow.
As seen above our playwright tests have passed since we are just checking for visibility of the elements but argos checks will fail, since the image logo height and width has been updated and there are some text updates. The summary will be added by argos-ci bot.
On clicking the inspect link available under build column, you will be navigated to the argos-ci dashboard build page wherein you will be able to view and review the diff changes captured as part of additional visual checks compared with the baseline image captured initially. This matches with the page changes that we had done like changing the height and width of the two logos and appending some extra text to the elements in the page. Since, the playwright test have passed trace is not available as we had specified trace: ‘retain-on-failure’.
Now, we will make some additional changes in the page.tsx file and change the link: Docs to Documents also I will add an extra assertion to the test that checks for the previous link name Docs to be available, which will cause an test failure since its renamed to Documents now.
import { argosScreenshot } from "@argos-ci/playwright"
import { expect, test } from "@playwright/test"
test("Verify home page designs", async ({ page }) => {
await page.goto("/")
await expect(page.getByAltText('Vercel Logo')).toBeVisible()
await expect(page.getByAltText('Next.js Logo')).toBeVisible()
await expect(page.getByText("Docs")).toBeVisible()
await argosScreenshot(page, "Homepage")
})
Commit and push the changes to remote repository which will fail the playwright tests and argos checks as well.
On checking the build in argos-ci you will see the below details, since the test has failed functionally, it has no baselines to compare with additionally we will see on test failure playwright trace available in build page which will open up the trace file for debugging purposes from argos itself.
Next, I will update my tests to check for Documents link and will push the new changes to remote repository and now our playwright tests will pass but the argos checks will fail again for the changes: height and width of the two images, additional text appended for few elements and Docs being renamed to Documents which does not match per baseline image.
To review the snapshot changes, click on the Review changes available in the top right corner of the build page you will see Approve and Reject changes options available.
I will go ahead and approve the latest changes from argos build and merge the pull request changes in the main branch.
Once the changes have been merged to main, a new Github actions workflow job will be triggered for playwright that will pass now and a new reference image build will be generated in argos-ci with the new changes that can be used for future comparisons.
This is how you can perform visual testing and verify the visual changes directly in CI using Github with Argos and Playwright integration. Hope you found this article to be helpful and do checkout their docs page for additional information. Thanks for reading!!
