CodeX
Published in

CodeX

How to Run Cypress Docker Images as Build Agents in Azure DevOps

Reduce test execution time with the flexibility to run against different browser versions

Authors:

Oliver Agar, Senior Software Development Engineer in Test Engineering at Sidetrade

Sam Berry, Software Development Engineer in Test Engineering at Sidetrade

Simon Parr, Senior Software Developer in Test Engineering at Sidetrade

Here at Sidetrade, we have been focusing on developing our Automated Tests to run in a build pipeline. Our mantra is to be able to run our tests at any time against any environment — be it Release or Development — so that we can deliver the best possible quality to our customers.

We explain below how we have managed to run our Automated UI tests in Azure DevOps along with running tests in parallel to reduce test execution time whilst also giving us the flexibility to run against different browser versions.

Setting up Azure DevOps Agents

Originally, we needed to install Cypress onto every agent in our pipeline. Not necessarily a problem since it is simply installed through Node Package Manager (npm). However, over time this took up resources on agents with every new version of Cypress.

With this in mind, we created docker images that contained the Cypress image and associated necessary tools (e.g. Oracle Instant Client). Agents would check out our tests and run them inside a docker container. Without the need to install the Cypress each run, agents were not spending time installing the necessary packages every run, in turn, speeding up test execution.

A diagram depicting the journey of setting up Azure DevOps agents as described in the previous paragraph
Image by author

However, if tests got stuck or took too long, Azure DevOps would stop the run, but the container image would still be running on the agent eating up its resources. We needed to go into the agent to kill the container.

We then looked into creating Containerised Docker Images for our Agents to use. This is a single image for our build agents to use and execute our tests. We now only need to maintain our image which can then be used across multiple agents.

A diagram depicting the journey of creating Containerised Docker Images for Agents as described in the text
Image by author

Two images were created:

· A base image with Edge, Chrome and Firefox based from Cypress Base Node Image

· Using our browser image, another image is built which has installed Cypress and other tools needed for our test

This gave us the flexibility to re-use the same image time and time again and the ability to scale our build agents easily. Using our custom image as the agent, at test execution it would copy in our tests and run the feature under test.

Browser version management

To adequately represent our app’s target users, we need to test on Chrome, Edge and Firefox. Each of these browsers release a new major version every six weeks approximately.

From experience, we have seen showstopping issues with the introduction of new browser features, particularly around the areas of security and cookie handling, so it is essential that we track the current release version of each browser closely by updating our base image every few months.

As we now have base images of various browser images, we can quickly run our tests on previous browser versions when a client has issues with a particular browser version.

We also track the beta branch of most browsers to ensure we get an early alert of any major issues.

Parallelisation of tests

As our test suites grew, we needed to reduce the time it takes for test execution. Whilst this is possible using Cypress Dashboard, this was not an option.

So, we leveraged the Matrix Strategy in Azure DevOps to farm out test spec/browser execution across the available agents.

parameters:- name: platformdisplayName: Platform Under Testtype: stringdefault: Platform01values:- Platform01- Platform02- Platform03- name: uiSuitesdisplayName: UI Feature Teststype: objectdefault:suites:- ‘Feature1’- ‘Feature2’- ‘Feature3’- ‘Feature4’- name: testBrowserdisplayName: Browsers to Runtype: objectdefault:browsers:- chrome- edge- firefoxvariables:dockerContainer: <Our Custom Cypress Docker Image>stages:- stage: uiRundependsOn: []condition: succeededOrFailed()jobs:- job: uiSuitestimeoutInMinutes: Xpool: Linuxcontainer: $[ variables[‘dockerContainer’] ]dependsOn: []strategy:matrix:${{ each suite in parameters.uiSuites.suites }}:${{ each browser in parameters.testBrowser.browsers }}:${{ suite }}-${{ browser }}:TAG: ${{ suite }}BROWSER: ${{ browser }}steps:- script: npm run cy:run:$(platform) run — — browser $(BROWSER) –spec “cypress/integration/$(TAG).featurecontinueOnError: true- task: PublishBuildArtifacts@1inputs:PathtoPublish: ‘$(System.DefaultWorkingDirectory)’ArtifactName: ‘drop_$(TAG)_$(BROWSER)’publishLocation: ‘Container’- task: PublishTestResults@2inputs:testResultsFormat: ‘JUnit’testResultsFiles: ‘**/test-results.*.xml’failTaskOnFailedTests: truetestRunTitle: ‘UI Run $(uiSuites) — $(BROWSER)’

This gives us the flexibility to select which platform to test against, features to run, and browsers to run against at runtime of the pipeline. For each feature/browser combination, a build job is created in Azure DevOps. When an agent becomes available from the pool, the feature test is run. The more agents are available, the more feature/browser combinations we can run in parallel. We also have scope to bring more agents online depending on the job load in Azure DevOps. Overall, this improved our test execution time by over 80%.

Diagram depicting running feature/browser combinations in parallel with additional agents
Image by author

Not all tests are suitable for parallelisation, either due to oversights in the app, our tests, or by design — for efficiency. To ensure that we include only suitable test suites in the parallelised runs we identified which of our suites would have problems if run in parallel and created a separate step in the pipeline to ensure these are each run on a single agent, termed ‘sequential suites’.

Conclusion

By using our custom docker build images as containerised agents coupled with Matrix Strategy, we have seen test execution time drop from around eight hours to just one hour. In order not to hog build agents from developer builds we have also introduced dedicated testing farm agents to execute our tests on.

Let us know what you think in the comments below, and don’t forget to give us a follow for more on what we’re up to here at the Sidetrade Tech Hub!

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Sidetrade Tech Hub

Sidetrade Tech Hub

Views from the software developers, data scientists and other tech experts at Sidetrade — the global AI-powered Order-To-Cash platform: www.sidetrade.com