How to publish Cypress reports to Azure pipelines
Why we use Cypress?
With Cypress, we can use the same framework for both UI and API automation testing. Comparing with other automation testing tools, such as Selenium, Cypress waits and retries automatically for commands and assertions, we don’t need to add explicit waits. Cypress runs tests in real browsers, we don’t need to download different versions of browser drivers.
Challenges we are facing
We integrate Cypress tests with Azure CI/CD pipelines and record tests to Cypress Cloud. With Cypress Cloud, we can easily integrate with Slack to post the regression results to different slack channels. Based on this setup, it makes easy for QA engineers to be notified of the testing results and check the screenshots/videos of failures.
But this is not the case for developers, they prefer to check the test results from CI/CD pipelines directly without checking slack messages or logging in to Cypress Cloud. Time is Money. :) So we need to find out how to publish Cypress test results to Azure pipelines.
How I made it work
Firstly, I did some research online regarding the task to publish test reports and what format of reports that Azure supports. Fortunately, Azure has the built-in task PublishTestResults to publish test results.
# Publish Test Results v2
# Publish test results to Azure Pipelines.
- task: PublishTestResults@2
inputs:
testResultsFormat: 'JUnit' # 'JUnit' | 'NUnit' | 'VSTest' | 'XUnit' | 'CTest'. Alias: testRunner. Required. Test result format. Default: JUnit.
testResultsFiles: '**/TEST-*.xml' # string. Required. Test results files. Default: **/TEST-*.xml.
#searchFolder: '$(System.DefaultWorkingDirectory)' # string. Search folder. Default: $(System.DefaultWorkingDirectory).
#mergeTestResults: false # boolean. Merge test results. Default: false.
#failTaskOnFailedTests: false # boolean. Fail if there are test failures. Default: false.
#testRunTitle: # string. Test run title.
# Advanced
#buildPlatform: # string. Alias: platform. Build Platform.
#buildConfiguration: # string. Alias: configuration. Build Configuration.
#publishRunAttachments: true # boolean. Upload test results files. Default: true.
This task supports the following formats: JUnit, NUnit, VSTest, XUnit, CTest. Apart from JUnit, other reports are mainly for .Net framework.
So the next step is to check whether Cypress supports JUnit report. The answer is YES.
Because Cypress is built on top of Mocha, that means any reporter built for Mocha can be used with Cypress. Here is a list of built in Mocha reporters.
By default, Cypress uses the
spec
reporter to output information toSTDOUT
.We’ve also added the two most common 3rd party reporters for Mocha. These are built into Cypress and you can use them without installing anything.
Finally, we support creating your own custom reporters or using any kind of 3rd party reporter.
Source: https://docs.cypress.io/guides/tooling/reporters
Since we want to generate multiple reporters, such as spec
and Junit
, we utilize npm package mocha-multi-reporters to achieve this.
Step 1: install the npm packages
npm i -D mocha-multi-reporters
npm i -D mocha-junit-reporter
Step 2: update Cypress config to use mocha-multi-reporters
reporter: 'mocha-multi-reporters',
reporterOptions: {
configFile: 'reporter-config.json',
},
Step 3: create reporter-config.json
to specify the reporters to be used
{
"reporterEnabled": "spec, mocha-junit-reporter",
"mochaJunitReporterReporterOptions": {
"mochaFile": "results/test-result-[hash].xml"
}
}
Step 4: add task in YAML file to publish the JUnit reporters
- task: PublishTestResults@2
displayName: 'Publish test results'
inputs:
testResultsFiles: 'test-result-*.xml'
searchFolder: '$(Build.Repository.LocalPath)/$(repo.repoName)/${{ parameters.testFolder }}/results'
mergeTestResults: true
failTaskOnFailedTests: true
continueOnError: true
condition: succeededOrFailed()
Now developers can simply check the results under theTests
tab from the CI/CD pipeline.
Bonus Point
Test reports are published to the CI/CD pipeline, but when we check the report, we can only see the error messages for failed tests without any screenshots or videos.
After spending a lot of time online, I found and updated the following powershell script to attach screenshots to the reports.
# Configuration
$global:cwd = Get-Location
$global:screenshotPath = Join-Path -Path $global:cwd -ChildPath "cypress/screenshots"
# Global Variables
$screenshotsHashtable = @{ } # Key = test name, Value = Full screenshot filename
# Authentication - Azure DevOps
$accessToken = $env:SYSTEM_ACCESSTOKEN
$teamFoundationCollectionUri = $env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI
$teamProjectId = $env:SYSTEM_TEAMPROJECTID
$buildId = $env:BUILD_BUILDID
$headers = @{ Authorization = "Bearer " + $accessToken }
# Functions
function Save-Screenshots-Hashtable {
Write-Host "Searching for screenshots in '$($global:screenshotPath)'"
$screenshots = Get-ChildItem -Path $screenshotPath -Filter *.png -Recurse -File | Select-Object -Property FullName, Name
foreach ($screenshot in $screenshots) {
$testName = $screenshot.Name -replace " --", ""
$testName = $testName -replace " \(failed\).png", ""
$screenshotsHashtable[$testName] = $screenshot.FullName
Write-Host "Found screenshot '$($screenshot)' for test '$($testName)'"
}
}
function Get-TestRuns {
$today = Get-Date
$tomorrow = $today.AddDays(1)
$yesterday = $today.AddDays(-1)
$minLastUpdatedDate = $yesterday.ToString("yyyy-MM-dd")
$maxLastUpdatedDate = $tomorrow.ToString("yyyy-MM-dd")
$testRunUrl = "$($teamFoundationCollectionUri)$($teamProjectId)/_apis/test/runs?api-version=5.1&minLastUpdatedDate=$($minLastUpdatedDate)&maxLastUpdatedDate=$($maxLastUpdatedDate)&buildIds=$($buildId)"
Write-Host "Getting Test Runs from '$testRunUrl'"
$testRunResponse = Invoke-RestMethod -Uri $testRunUrl -Headers $headers
$latestRun = $testRunResponse.value[$testRunResponse.count - 1];
Get-Test-Results -runId $latestRun.id
}
function Find-Screenshot-From-Test-Result {
param($testName)
Write-Host "Searching screenshot for test '$($testName)'"
$screenshotFilename = $screenshotsHashtable[$testName]
if ($null -ne $screenshotFilename) {
Write-Host -ForegroundColor:Green "Found screenshot '$($screenshotFilename)' matching test '$($testName)'"
}
else {
Write-Host -ForegroundColor:Red "No screenshot found matching test '$($name)'"
}
return $screenshotFilename
}
function Add-Attachment-To-TestResult {
param($screenshotFilename, $testResultId)
$screenshotFilenameWithoutPath = Split-Path $screenshotFilename -leaf
$createTestResultsAttachmentUrl = "$teamFoundationCollectionUri$teamProjectId/_apis/test/runs/$($runId)/results/$($testResultId)/attachments?api-version=5.1-preview&outcomes=Failed"
$base64string = [Convert]::ToBase64String([IO.File]::ReadAllBytes($screenshotFilename))
$body = @{
fileName = $screenshotFilenameWithoutPath
comment = "Attaching screenshot"
attachmentType = "GeneralAttachment"
stream = $base64string
}
$json = $body | ConvertTo-Json
Write-Host "Attaching screenshot by posting to '$($createTestResultsAttachmentUrl)'"
$response = Invoke-RestMethod $createTestResultsAttachmentUrl -Headers $headers -Method Post -Body $json -ContentType "application/json"
Write-Host "Response from posting screenshot '$($response)'"
}
function Get-Test-Results {
param($runId)
$testResultsUrl = "$teamFoundationCollectionUri$teamProjectId/_apis/test/runs/$($runId)/results?api-version=5.1&outcomes=Failed"
Write-Host "Getting Test Results from '$($testResultsUrl)'"
$testResultsResponse = Invoke-RestMethod -Uri $testResultsUrl -Headers $headers
foreach ($testResult in $testResultsResponse.value) {
Write-Host "Found failing test '$($testResult.testCase.name)'"
$screenshotFilename = Find-Screenshot-From-Test-Result -testName ($testResult.testCase.name -replace '"', "")
if ($null -ne $screenshotFilename) {
Add-Attachment-To-TestResult -screenshotFilename $screenshotFilename -testResultId $testResult.id
}
}
}
# Entry Point
Write-Host "TeamFoundationCollectionUri: $teamFoundationCollectionUri"
Write-Host "TeamProjectId: $teamProjectId"
Write-Host "BuildId: $buildId"
Write-Host ""
Save-Screenshots-Hashtable
Get-TestRuns
Conclusion
The above setup can save time for deployment especially for developers, they can simply get the clear reports for the regression tests in one place, if all tests pass, they can make the decision quickly to release changes to Production.
If there are any failed tests, we can check the results and screenshots, but such information is not very clear comparing with the videos captured in Cypress Cloud. If your organization has enough licenses, I would suggest to give developers access to Cypress Cloud so that they can check and debug the failures much more easily.
Moreover, with Cypress Cloud’s powerful integrations, you can integrate Cypress Cloud with GitHub, so that developers will have the ability to check the regression testing results from PR in GitHub directly. You can also configure the integration in Cypress Cloud to add PR comments based on the results.
PS: I will create another story for GitHub integration with Cypress Cloud in the next few weeks. Happy testing!