What is Cypress Cloud?
Cypress Cloud is a powerful tool to record Cypress tests with screenthots and videos to the cloud. Also with the latest version of Cypress app, we can config it to receive desktop notifications for testing results and debug the failed tests which are executed in CI from Cypress app locally.
Challenges with Recording Tests
In order to record the tests to Cypress Cloud, we need to pay for the number of tests which are recorded. Recently I found that our developers are recording tests in Demo environment to Cypress Cloud and re-running those tests for multiple times and this is not a good practice since since we have Staging environment to run regression tests before releasing new features to Production.
Exploring Parallel Exeuction Options
After catching up with developers, we made the decision to run tests in Demo without recording to Cypress Cloud since Demo is only for feature implementation and showcase.
Firstly, I though I can simply remove --record --key from the command and still run tests in parallel, but this is not working. If we want to run Cypress tests in parallel, we have to record the tests to Cypress Cloud with paramater like — ci-build-id . Cypress will use this build id to distribute the specs to multiple agents.
npx cypress run --browser chrome --record --key $(CYPRESS_KEY) --parallel --tag $(TAG) --ci-build-id Build-$(Build.BuildId)-$(System.JobAttempt) --group Build-$(Build.BuildId)-$(System.JobAttempt)After doing some research online, I found the following two npm packages which can meet our need.
cypress-parallel is a powerful tool to run tests in multiple threads, but it doesn’t support CI and we have to create npm script for running Cypress tests first, also pass the specs folder to the following command. If the test agent has limitted resource, such as virtual CPUs, it won’t save the execution time significantly.
cypress-parallel -s cy:run -t 2 -d <your-cypress-specs-folder>cypress-split is the tool which I chose to implement the parallel execution from CI. To begin with, I just followed the steps and updated the command in YAML template as following, but it didn’t work at the first try.
npx cypress run --env split=true
In the above document, the author provided examples for some CI tools, but we are using Azure DevOps as our CI tool which is not included in the examples.
Implementing cypress-split in Azure DevOps
After digging into the code of cypress-split , I found that this plugin relies on two environment variables to split the specs: SPLIT and SPLIT_INDEX .
Based on the understanding of the above piece of code, I updated our YAML template to pass in these two environment variables to the step for running Cypress tests and it works perfectly. :)
- pwsh: |
$index = $(System.JobPositionInPhase) - 1
echo "##vso[task.setvariable variable=index;isOutput=true]$index"
name: index
displayName: "Get Job Index"
- pwsh: |
npx nx cy:run ${{ parameters.app }} -- --env split=true --browser chrome ${{ parameters.cypressCLIOptions }}
workingDirectory: "$(Build.Repository.LocalPath)/$(repo.repoName)/${{ parameters.testFolder }}"
displayName: "Execute tests"
env:
SPLIT: $(AGENTS)
SPLIT_INDEX: $(index.index)AGENTS is the pipeline variable to specify number of agents to run the tests.
SPLIT_INDEX should be starting from 0 as the name indicated that it is an index. It took me sometime to find the related pipeline variable for Azure to make it work and fortunately I found that I can use System.JobPositionInPhase . But it starts from 1, so I have to use the above powershell script to update the value and save it to another variable.
Conclusion
Even though I managed to make it work for running Cypress tests in parallel without recording to Cypress Cloud, I still prefer to record tests to Cypress Cloud which makes it easier to debug the failing tests, re-run the failing tests locally and etc.
