Cypress

How to publish Cypress reports to Azure pipelines

Why we use Cypress?

With Cypress, we can use the same framework for both UI and API automation testing. Comparing with other automation testing tools, such as Selenium, Cypress waits and retries automatically for commands and assertions, we don’t need to add explicit waits. Cypress runs tests in real browsers, we don’t need to download different versions of browser drivers.

Challenges we are facing

We integrate Cypress tests with Azure CI/CD pipelines and record tests to Cypress Cloud. With Cypress Cloud, we can easily integrate with Slack to post the regression results to different slack channels. Based on this setup, it makes easy for QA engineers to be notified of the testing results and check the screenshots/videos of failures.

But this is not the case for developers, they prefer to check the test results from CI/CD pipelines directly without checking slack messages or logging in to Cypress Cloud. Time is Money. :) So we need to find out how to publish Cypress test results to Azure pipelines.

How I made it work

Firstly, I did some research online regarding the task to publish test reports and what format of reports that Azure supports. Fortunately, Azure has the built-in task PublishTestResults to publish test results.

# Publish Test Results v2
# Publish test results to Azure Pipelines.
- task: PublishTestResults@2
  inputs:
    testResultsFormat: 'JUnit' # 'JUnit' | 'NUnit' | 'VSTest' | 'XUnit' | 'CTest'. Alias: testRunner. Required. Test result format. Default: JUnit.
    testResultsFiles: '**/TEST-*.xml' # string. Required. Test results files. Default: **/TEST-*.xml.
    #searchFolder: '$(System.DefaultWorkingDirectory)' # string. Search folder. Default: $(System.DefaultWorkingDirectory).
    #mergeTestResults: false # boolean. Merge test results. Default: false.
    #failTaskOnFailedTests: false # boolean. Fail if there are test failures. Default: false.
    #testRunTitle: # string. Test run title. 
  # Advanced
    #buildPlatform: # string. Alias: platform. Build Platform. 
    #buildConfiguration: # string. Alias: configuration. Build Configuration. 
    #publishRunAttachments: true # boolean. Upload test results files. Default: true.

This task supports the following formats: JUnit, NUnit, VSTest, XUnit, CTest. Apart from JUnit, other reports are mainly for .Net framework.

So the next step is to check whether Cypress supports JUnit report. The answer is YES.

Because Cypress is built on top of Mocha, that means any reporter built for Mocha can be used with Cypress. Here is a list of built in Mocha reporters.

Mocha’s built-in reporters

By default, Cypress uses the spec reporter to output information to STDOUT.

We’ve also added the two most common 3rd party reporters for Mocha. These are built into Cypress and you can use them without installing anything.

teamcity

junit

Finally, we support creating your own custom reporters or using any kind of 3rd party reporter.

Source: https://docs.cypress.io/guides/tooling/reporters

Since we want to generate multiple reporters, such as specand Junit , we utilize npm package mocha-multi-reporters to achieve this.

Step 1: install the npm packages

npm i -D mocha-multi-reporters
npm i -D mocha-junit-reporter

Step 2: update Cypress config to use mocha-multi-reporters

Cypress config example

reporter: 'mocha-multi-reporters',
reporterOptions: {
  configFile: 'reporter-config.json',
},

Step 3: create reporter-config.json to specify the reporters to be used

{
  "reporterEnabled": "spec, mocha-junit-reporter",
  "mochaJunitReporterReporterOptions": {
    "mochaFile": "results/test-result-[hash].xml"
  }
}

Step 4: add task in YAML file to publish the JUnit reporters

- task: PublishTestResults@2
    displayName: 'Publish test results'
    inputs:
      testResultsFiles: 'test-result-*.xml'
      searchFolder: '$(Build.Repository.LocalPath)/$(repo.repoName)/${{ parameters.testFolder }}/results'
      mergeTestResults: true
      failTaskOnFailedTests: true
    continueOnError: true
    condition: succeededOrFailed()

Now developers can simply check the results under theTests tab from the CI/CD pipeline.

Bonus Point

Test reports are published to the CI/CD pipeline, but when we check the report, we can only see the error messages for failed tests without any screenshots or videos.

After spending a lot of time online, I found and updated the following powershell script to attach screenshots to the reports.

# Configuration
$global:cwd = Get-Location
$global:screenshotPath = Join-Path -Path $global:cwd -ChildPath "cypress/screenshots"

# Global Variables
$screenshotsHashtable = @{ } # Key = test name, Value = Full screenshot filename

# Authentication - Azure DevOps

$accessToken = $env:SYSTEM_ACCESSTOKEN
$teamFoundationCollectionUri = $env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI
$teamProjectId = $env:SYSTEM_TEAMPROJECTID
$buildId = $env:BUILD_BUILDID
$headers = @{ Authorization = "Bearer " + $accessToken }

# Functions

function Save-Screenshots-Hashtable {
 Write-Host "Searching for screenshots in '$($global:screenshotPath)'"

 $screenshots = Get-ChildItem -Path $screenshotPath  -Filter *.png -Recurse -File | Select-Object -Property FullName, Name
 
 foreach ($screenshot in $screenshots) {

  $testName = $screenshot.Name -replace " --", ""
  $testName = $testName -replace " \(failed\).png", ""

  $screenshotsHashtable[$testName] = $screenshot.FullName

  Write-Host "Found screenshot '$($screenshot)' for test '$($testName)'"
 }
}

function Get-TestRuns {

 $today = Get-Date
 $tomorrow = $today.AddDays(1)
 $yesterday = $today.AddDays(-1)

 $minLastUpdatedDate = $yesterday.ToString("yyyy-MM-dd")
 $maxLastUpdatedDate = $tomorrow.ToString("yyyy-MM-dd")

 $testRunUrl = "$($teamFoundationCollectionUri)$($teamProjectId)/_apis/test/runs?api-version=5.1&minLastUpdatedDate=$($minLastUpdatedDate)&maxLastUpdatedDate=$($maxLastUpdatedDate)&buildIds=$($buildId)"
 Write-Host "Getting Test Runs from '$testRunUrl'"

 $testRunResponse = Invoke-RestMethod -Uri $testRunUrl -Headers $headers

 $latestRun = $testRunResponse.value[$testRunResponse.count - 1];

 Get-Test-Results -runId $latestRun.id
}

function Find-Screenshot-From-Test-Result {
 param($testName)

 Write-Host "Searching screenshot for test '$($testName)'"

 $screenshotFilename = $screenshotsHashtable[$testName]

 if ($null -ne $screenshotFilename) {
  Write-Host -ForegroundColor:Green "Found screenshot '$($screenshotFilename)' matching test '$($testName)'"
 }
  else {
  Write-Host -ForegroundColor:Red "No screenshot found matching test '$($name)'"
 }

 return $screenshotFilename
}

function Add-Attachment-To-TestResult {
 param($screenshotFilename, $testResultId)

 $screenshotFilenameWithoutPath = Split-Path $screenshotFilename -leaf

 $createTestResultsAttachmentUrl = "$teamFoundationCollectionUri$teamProjectId/_apis/test/runs/$($runId)/results/$($testResultId)/attachments?api-version=5.1-preview&outcomes=Failed"

 $base64string = [Convert]::ToBase64String([IO.File]::ReadAllBytes($screenshotFilename))

 $body = @{
  fileName       = $screenshotFilenameWithoutPath
  comment        = "Attaching screenshot"
  attachmentType = "GeneralAttachment"
  stream         = $base64string
 }

 $json = $body | ConvertTo-Json

 Write-Host "Attaching screenshot by posting to '$($createTestResultsAttachmentUrl)'"

 $response = Invoke-RestMethod $createTestResultsAttachmentUrl -Headers $headers -Method Post -Body $json -ContentType "application/json"

 Write-Host "Response from posting screenshot '$($response)'"
}

function Get-Test-Results {
 param($runId)

 $testResultsUrl = "$teamFoundationCollectionUri$teamProjectId/_apis/test/runs/$($runId)/results?api-version=5.1&outcomes=Failed"

 Write-Host "Getting Test Results from '$($testResultsUrl)'"

 $testResultsResponse = Invoke-RestMethod -Uri $testResultsUrl -Headers $headers

 foreach ($testResult in $testResultsResponse.value) {

  Write-Host "Found failing test '$($testResult.testCase.name)'"

  $screenshotFilename = Find-Screenshot-From-Test-Result -testName ($testResult.testCase.name -replace '"', "")

  if ($null -ne $screenshotFilename) {
   Add-Attachment-To-TestResult -screenshotFilename $screenshotFilename -testResultId $testResult.id
  }
 }
}

# Entry Point
Write-Host "TeamFoundationCollectionUri: $teamFoundationCollectionUri"
Write-Host "TeamProjectId: $teamProjectId"
Write-Host "BuildId: $buildId"
Write-Host ""

Save-Screenshots-Hashtable
Get-TestRuns

Conclusion

The above setup can save time for deployment especially for developers, they can simply get the clear reports for the regression tests in one place, if all tests pass, they can make the decision quickly to release changes to Production.

If there are any failed tests, we can check the results and screenshots, but such information is not very clear comparing with the videos captured in Cypress Cloud. If your organization has enough licenses, I would suggest to give developers access to Cypress Cloud so that they can check and debug the failures much more easily.

Moreover, with Cypress Cloud’s powerful integrations, you can integrate Cypress Cloud with GitHub, so that developers will have the ability to check the regression testing results from PR in GitHub directly. You can also configure the integration in Cypress Cloud to add PR comments based on the results.

PS: I will create another story for GitHub integration with Cypress Cloud in the next few weeks. Happy testing!