How to Run Cypress Docker Images as Build Agents in Azure DevOps

Reduce test execution time with the flexibility to run against different browser versions

Authors:

Oliver Agar, Senior Software Development Engineer in Test Engineering at Sidetrade

Sam Berry, Software Development Engineer in Test Engineering at Sidetrade

Simon Parr, Senior Software Developer in Test Engineering at Sidetrade

Here at Sidetrade, we have been focusing on developing our Automated Tests to run in a build pipeline. Our mantra is to be able to run our tests at any time against any environment — be it Release or Development — so that we can deliver the best possible quality to our customers.

We explain below how we have managed to run our Automated UI tests in Azure DevOps along with running tests in parallel to reduce test execution time whilst also giving us the flexibility to run against different browser versions.

Setting up Azure DevOps Agents

Originally, we needed to install Cypress onto every agent in our pipeline. Not necessarily a problem since it is simply installed through Node Package Manager (npm). However, over time this took up resources on agents with every new version of Cypress.

With this in mind, we created docker images that contained the Cypress image and associated necessary tools (e.g. Oracle Instant Client). Agents would check out our tests and run them inside a docker container. Without the need to install the Cypress each run, agents were not spending time installing the necessary packages every run, in turn, speeding up test execution.

Image by author

However, if tests got stuck or took too long, Azure DevOps would stop the run, but the container image would still be running on the agent eating up its resources. We needed to go into the agent to kill the container.

We then looked into creating Containerised Docker Images for our Agents to use. This is a single image for our build agents to use and execute our tests. We now only need to maintain our image which can then be used across multiple agents.

Image by author

Two images were created:

· A base image with Edge, Chrome and Firefox based from Cypress Base Node Image

· Using our browser image, another image is built which has installed Cypress and other tools needed for our test

This gave us the flexibility to re-use the same image time and time again and the ability to scale our build agents easily. Using our custom image as the agent, at test execution it would copy in our tests and run the feature under test.

Browser version management

To adequately represent our app’s target users, we need to test on Chrome, Edge and Firefox. Each of these browsers release a new major version every six weeks approximately.

From experience, we have seen showstopping issues with the introduction of new browser features, particularly around the areas of security and cookie handling, so it is essential that we track the current release version of each browser closely by updating our base image every few months.

As we now have base images of various browser images, we can quickly run our tests on previous browser versions when a client has issues with a particular browser version.

We also track the beta branch of most browsers to ensure we get an early alert of any major issues.

Parallelisation of tests

As our test suites grew, we needed to reduce the time it takes for test execution. Whilst this is possible using Cypress Dashboard, this was not an option.

So, we leveraged the Matrix Strategy in Azure DevOps to farm out test spec/browser execution across the available agents.

parameters:
- name: platform
displayName: Platform Under Test
type: string
default: Platform01
values:
- Platform01
- Platform02
- Platform03
- name: uiSuites
displayName: UI Feature Tests
type: object
default:
suites:
- ‘Feature1’
- ‘Feature2’
- ‘Feature3’
- ‘Feature4’
- name: testBrowser
displayName: Browsers to Run
type: object
default:
browsers:
- chrome
- edge
- firefox
variables:
dockerContainer: <Our Custom Cypress Docker Image>
stages:
- stage: uiRun
dependsOn: []
condition: succeededOrFailed()
jobs:
- job: uiSuites
timeoutInMinutes: X
pool: Linux
container: $[ variables[‘dockerContainer’] ]
dependsOn: []
strategy:
matrix:
${{ each suite in parameters.uiSuites.suites }}:
${{ each browser in parameters.testBrowser.browsers }}:
${{ suite }}-${{ browser }}:
TAG: ${{ suite }}
BROWSER: ${{ browser }}
steps:
- script: npm run cy:run:$(platform) run — — browser $(BROWSER) –spec “cypress/integration/$(TAG).feature
continueOnError: true
- task: PublishBuildArtifacts@1
inputs:
PathtoPublish: ‘$(System.DefaultWorkingDirectory)’
ArtifactName: ‘drop_$(TAG)_$(BROWSER)’
publishLocation: ‘Container’
- task: PublishTestResults@2
inputs:
testResultsFormat: ‘JUnit’
testResultsFiles: ‘**/test-results.*.xml’
failTaskOnFailedTests: true
testRunTitle: ‘UI Run $(uiSuites) — $(BROWSER)’

This gives us the flexibility to select which platform to test against, features to run, and browsers to run against at runtime of the pipeline. For each feature/browser combination, a build job is created in Azure DevOps. When an agent becomes available from the pool, the feature test is run. The more agents are available, the more feature/browser combinations we can run in parallel. We also have scope to bring more agents online depending on the job load in Azure DevOps. Overall, this improved our test execution time by over 80%.

Image by author

Not all tests are suitable for parallelisation, either due to oversights in the app, our tests, or by design — for efficiency. To ensure that we include only suitable test suites in the parallelised runs we identified which of our suites would have problems if run in parallel and created a separate step in the pipeline to ensure these are each run on a single agent, termed ‘sequential suites’.

Conclusion

By using our custom docker build images as containerised agents coupled with Matrix Strategy, we have seen test execution time drop from around eight hours to just one hour. In order not to hog build agents from developer builds we have also introduced dedicated testing farm agents to execute our tests on.

Let us know what you think in the comments below, and don’t forget to give us a follow for more on what we’re up to here at the Sidetrade Tech Hub!