Open in app

Sign in

Write

Sign in

Annotate ADO PR with checkov results

Benjamin Garside
Version 1
Published in
6 min readJul 16, 2024

Being able to run all sorts of checks and validation in a pipeline that runs on the creation of a pull request (PR) is great, but it can then be hard to find out what the output of these checks are without diving in to the details of the pipeline run. A simple solution to this is to make use of the comment system within the PR to annotate them with easy to digest summary results.

In this post I am going to look at running checkov scans and showing how we can surface the results in a way that helps a reviewer of a PR see what they need quickly.

Running the scan

In this example we are going to run the checkov scan via their docker image in a simple YAML pipeline:

- bash: |
    docker pull bridgecrew/checkov              
  workingDirectory: $(Agent.BuildDirectory)/component
  displayName: "Docker pull bridgecrew/checkov"
- bash: |
    docker run \
      --volume $(pwd):/tf bridgecrew/checkov \
      --directory /tf \
      --output junitxml \
      --soft-fail > $(pwd)/CheckovReport.xml              
  workingDirectory: $(System.DefaultWorkingDirectory)
  displayName: "Run Checkov"

As you can see from that snippet I am pulling the image then running a scan against my default working directory. The key part of this is that it has soft-fail set and that we are outputting in a JUnit format to a file called CheckovReport.xml. This means that even if the checks fail, the step will pass and the file will always be generated.

Viewing the results

We can use ADOs inbuilt ability to read and manage JUnit test results by uploading the xml file to the pipline with the follwoing task:

- task: PublishTestResults@2
  inputs:
    testRunTitle: "Checkov Results"
    failTaskOnFailedTests: true
    testResultsFormat: "JUnit"
    testResultsFiles: "CheckovReport.xml"
    searchFolder: "$(System.DefaultWorkingDirectory)"
  displayName: "Publish Checkov results"

Because we have failTaskOnFailedTest set to true this task will now fail the pipeline when checkov finds any errors. These test reults can now be viewed in the pipeline run, under the tests tab:

ADO pipeline test result view

This is great functionality, and it makes managing the errors across multiple builds really easy, however if we ran this via a PR, this is what we see:

Checkov pipeline failure viewed in a PR

All we know is that the step that publishes the test results has failed, and shows the following error:

There are one or more test failures detected in result files. Detailed summary of published test results can be viewed in the Tests tab.

This is not super helpful as we would have to go to that build from the pull request and then go to the tests tab.

Getting the results into the PR

Using the Pull Request Thread Comments Rest API we can use the information from the JUnit XML file and post a comment with the key information we need directly in to the PR as a comment!

First, we need to take a look at the structure of the JUnit XML file, here is an example of a scan I have previously run:

<?xml version="1.0" ?>
<testsuites disabled="0" errors="0" failures="2" tests="179" time="0.0">
 <testsuite disabled="0" errors="0" failures="2" name="terraform_plan scan" skipped="2" tests="179" time="0">
  <testcase name="[NONE][CKV_AZURE_12] Ensure that Network Security Group Flow Log retention period is 'greater than 90 days'" classname="/tfplan.json.azurerm_network_watcher_flow_log.main[&quot;nsg-uks-prd-dsh-tst-endpoint&quot;]" file="/tfplan.json"/>
  ...
  <testcase name="[NONE][CKV2_AZURE_41] Ensure storage account is configured with SAS expiration policy" classname="/tfplan.json.module.hub_storage_account.azurerm_storage_account.account[&quot;saotuksprddshtst01&quot;]" file="/tfplan.json">
   <failure type="failure" message="Ensure storage account is configured with SAS expiration policy">
Resource: module.hub_storage_account.azurerm_storage_account.account[&quot;saotuksprddshtst01&quot;]
File: /tfplan.json: 0-0
Guideline: https://docs.prismacloud.io/en/enterprise-edition/policy-reference/azure-policies/azure-iam-policies/bc-azure-2-41</failure>
  </testcase>
  <testcase name="[NONE][CKV2_AZURE_40] Ensure storage account is not configured with Shared Key authorization" classname="/tfplan.json.module.hub_storage_account.azurerm_storage_account.account[&quot;saotuksprddshtst01&quot;]" file="/tfplan.json">
   <failure type="failure" message="Ensure storage account is not configured with Shared Key authorization">
Resource: module.hub_storage_account.azurerm_storage_account.account[&quot;saotuksprddshtst01&quot;]
File: /tfplan.json: 0-0
Guideline: https://docs.prismacloud.io/en/enterprise-edition/policy-reference/azure-policies/azure-iam-policies/bc-azure-2-40</failure>
  </testcase>
  <testcase name="[NONE][CKV2_AZURE_1] Ensure storage for critical data are encrypted with Customer Managed Key" classname="/tfplan.json.module.hub_storage_account.azurerm_storage_account.account[&quot;saotuksprddshtst01&quot;]" file="/tfplan.json">
   <skipped type="skipped" message="TEMPORARY: This needs implimenting"/>
  </testcase>
  <testcase name="[NONE][CKV2_AZURE_31] Ensure VNET subnet is configured with a Network Security Group (NSG)" classname="/tfplan.json.module.vdc.module.subnets.azurerm_subnet.main[&quot;AzureFirewallSubnet&quot;]" file="/tfplan.json">
   <skipped type="skipped" message="TEMPORARY: This needs implimenting"/>
  </testcase>
 </testsuite>
</testsuites>

We can get most of the information we need in the testsuite node:

<testsuite disabled="0" errors="0" failures="2" name="terraform_plan scan" skipped="2" tests="179" time="0">

For this post we are going to use PowerShell to create the script which is going to do all the heavy lifting for us. We start by reading the XML file as an XML object and setting the key data:

[xml]$results = Get-Content CheckovReport.xml

$name = $results.testsuites.testsuite.name
$failures = $results.testsuites.testsuite.failures
$tests = $results.testsuites.testsuite.tests
$skipped = $results.testsuites.testsuite.skipped

As we are going to post our summary to a PR we are going to create a request object that we can append comments to, this includes a summary of the number of tests run, the number skipped and the number failed:

$request = @{ comments = New-Object Collections.ArrayList; status = "active" }
$comment = @{ content = "**Checkov Scan '$name'**`r`n`r`n:runner: Tests run: $tests`r`n:no_entry_sign: Skipped: $skipped`r`n:x: Failed: $failures`r`n`r`n"; commentType = "text" }

Next we will check this data to see if any checkov tests have been skipped, if they have we will loop over them and for each we will append a comment that will contain the test that was skipped, where it was skipped and the comment.

if($skipped -gt 0) {
    $comment.content += "The following checks have been skipped:`r`n"
    foreach ($testcase in $results.testsuites.testsuite.testcase) {
        if($null -ne $testcase.SelectSingleNode("./skipped")) {
            $comment.content += ":no_entry_sign: $($testcase.name) - $($testcase.classname) [$($testcase.skipped.message)]`r`n"
        }
    }
}

We can then do the same checking if the number of failures is greater than 0. The other difference here is that if there are no failures, we are setting the status of the PR comment to closed and active if there are failures. This means that if there are skipped tests but no failures it will not stop the PR from completing without someone acknowledging the failures.

if($failures -eq 0) {
    $request.status = "closed"
}
else {
    $comment.content +=  "`r`nThe following issues have been reported:`r`n"
    $request.status = "active"
    foreach ($testcase in $results.testsuites.testsuite.testcase) {
        if($null -ne ($testcase.SelectSingleNode("./failure"))) {
            $failure = Get-failureDetails($testcase.failure.InnerText)

            $comment.content += ":x: $($testcase.name) - $($failure.Groups['resource']) [guideline]($($failure.Groups['guideline']))`r`n"
        }
    }
    $comment.content += "`r`nThis comment thread is active, don't forget to mark it as resolved!"
}

Just like the skipped comment, we are posting the check that failed and where it failed, but we are also posting the link to the checkov guidelines so its super easy for a reviewer of the PR to get more information about the failure.

Now its just a case of adding the comment array to the request object and posting to the REST API endpoint, the full script looks like this:

Param(
    [string]$Path
)

[xml]$results = Get-Content $Path
$name = $results.testsuites.testsuite.name
$failures = $results.testsuites.testsuite.failures
$tests = $results.testsuites.testsuite.tests
$skipped = $results.testsuites.testsuite.skipped

function Get-failureDetails {
    param($failureString)
    $re = "Resource: (?<resource>.*)\nFile: (?<file>.*)\nGuideline: (?<guideline>.*)"
    return [regex]::Matches($failureString, $re)
}

$request = @{ comments = New-Object Collections.ArrayList; status = "active" }

$comment = @{ content = "**Checkov Scan '$name'**`r`n`r`n:runner: Tests run: $tests`r`n:no_entry_sign: Skipped: $skipped`r`n:x: Failed: $failures`r`n`r`n"; commentType = "text" }
if($skipped -gt 0) {
    $comment.content += "The following checks have been skipped:`r`n"
    foreach ($testcase in $results.testsuites.testsuite.testcase) {
        if($null -ne $testcase.SelectSingleNode("./skipped")) {
            $comment.content += ":no_entry_sign: $($testcase.name) - $($testcase.classname) [$($testcase.skipped.message)]`r`n"
        }
    }
}
if($failures -eq 0) {
    $request.status = "closed"
}
else {
    $comment.content +=  "`r`nThe following issues have been reported:`r`n"
    $request.status = "active"
    foreach ($testcase in $results.testsuites.testsuite.testcase) {
        if($null -ne ($testcase.SelectSingleNode("./failure"))) {
            $failure = Get-failureDetails($testcase.failure.InnerText)

            $comment.content += ":x: $($testcase.name) - $($failure.Groups['resource']) [guideline]($($failure.Groups['guideline']))`r`n"
        }
    }
    $comment.content += "`r`nThis comment thread is active, don't forget to mark it as resolved!"
}

$request.comments.Add($comment) | Out-Null
$url = "${env:SYSTEM_TEAMFOUNDATIONSERVERURI}${env:SYSTEM_TEAMPROJECT}/_apis/git/repositories/${env:BUILD_REPOSITORY_ID}/pullRequests/${env:SYSTEM_PULLREQUEST_PULLREQUESTID}/threads?api-version=5.1"
Invoke-RestMethod -Method "POST" -Uri $url -Body ($request | ConvertTo-Json) -ContentType "application/json" -Headers @{ Authorization = "Bearer ${env:SYSTEM_ACCESSTOKEN}" }

Now its just a case of adding a call to this script in the pipeline:

- task: PowerShell@2
  displayName: 'Comment PR with Checkov TF results'
  condition: and(or(succeeded(),failed()), eq(variables['Build.Reason'], 'PullRequest'))
  inputs:
    filePath: "$(System.DefaultWorkingDirectory)/prAnnotate_JUnit.ps1"
    arguments: '-Path $(System.DefaultWorkingDirectory)/CheckovReport.xml'
    workingDirectory: '$(System.DefaultWorkingDirectory)'
  env:
    SYSTEM_ACCESSTOKEN: $(System.AccessToken)

As you can see we have added a condition to this task, this is so that firstly, it will run regardless of if the previous task (the test result publish) fails or not, and also it will only run if the pipeline was triggered by a PR build validation.

Now when this is run we will get the following posted as a comment directly in to our PR:

PR comment with checkov summary

I hope that you would agree that this makes it much easier for a reviewer to get all the information that they need in order to review the output of the checkov scan and allows a place where they could also reply and discuss the results all neatly in one place.

About the Author:
Benjamin Garside is a Senior Azure DevOps Engineer at Version 1.

DevOps
Azure
Checkov
Powershell

--

--

Benjamin Garside
Version 1

Written by Benjamin Garside

2 Followers
Writer for

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams