How to utilize GitHub Code Scanning for Azure DevOps Repositories using Azure Pipelines, the classic editor

GitHub Code scanning is a powerful tool that can be utilized to find vulnerabilities and possible optimizations within your code. Currently, this feature is only available for codes that are stored in the GitHub repository, and is said to come to Azure DevOps sometime in the future (Source: devblogs.microsoft.com). However, there is a workaround that allows such a feature to be used for codes stored in Azure Repository as of now, and in this article, I’m going to tell you the steps needed to do it.

Photo by Agence Olloweb on Unsplash

To clarify, this workaround basically works by mirroring your current repository in Azure DevOps to the one in GitHub and requires you to have a subscription of GitHub Advanced Security if you wish to keep the repository private (Source: community.spiceworks.com). The benefit of doing this is whenever there is a new commit to the code in your repository, the update will automatically be pushed to your GitHub repository. With that out of the way, here is how you can do just that.

• First of all, we would be utilizing the GitHub API a lot, so you need to generate a GitHub Personal Access Token so that the pipeline has read and write access to your repository.
  - (GitHub PAT: docs.github.com)
  Side Note: To hit GitHub’s API, we won’t be using Azure’s built-in REST API Call in an agentless job, instead we’re going to use PowerShell tasks so that we could easily manage the sequence of tasks in an agent job since it would be more practical to execute in the long run.
• We would then provide the generated PAT, along with our GitHub username by setting it to a pipeline variable. In this case, we’ll set the variable name to GithubPAT and GithubUsername

Photo by Author

• Add an Agent job and open ‘Additional options’ to enable ‘Allow scripts to access the OAuth token’. Then, add 5 PowerShell tasks to the agent job, so that we could run the following steps:

Photo by Author

• In each of the PowerShell tasks, select the ‘Inline’ type, and enter these scripts:
• [Create a New GitHub repo]
  The following script will create a new private GitHub repository with the same name as the one in Azure DevOps, and the try-catch will ensure that the task will continue even if the repository already exists.

# URL for creating a repo using Github API
$url = "https://api.github.com/user/repos"

# Create header with PAT
$token = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes("$(GithubPAT)"))
$header = @{authorization = "Basic $token"}

$projectJSON = @{name = "$(Build.Repository.Name)"
               private = true
            } | ConvertTo-Json

try {
    Invoke-RestMethod -Uri $url -Method Post -ContentType "application/json" -Headers $header -Body ($projectJSON )
} catch {
    # Catch exception if repo already exist
    if ($_.Exception.Response.StatusCode.value__ -ne 422) { throw $_.Exception }
}

• [Mirror Git Repositories]
  Next, we’re going to mirror the content of the Azure Repository to the one in GitHub using the following script:

# Set Azure Repo URL
$AzurePAT = "$(System.AccessToken)"
$repoUri = "$(Build.Repository.Uri)"
$pattern = '.*@'
$azureurl = "https://$AzurePAT@" + ($repoUri -replace $pattern, '')

# Set GitHub Repo URL
$repoName = "$(Build.Repository.Name)"
$githuburl = "https://$(GithubPAT)@github.com/$(GithubUsername)/" + $repoName + '.git'

# Create a temporary copying folder
mkdir copy
Set-Location "$(Build.SourcesDirectory)/copy"
git clone $azureurl

# Change origin URL
Set-Location "$(Build.SourcesDirectory)/copy/$(Build.Repository.Name)/"
git remote rm origin
git remote add origin $githuburl

# Fetch and pull updated repo
git push origin --all -f

• [Add CodeQL to GitHub Repo]
  To add GitHub workflow to a repository, we would need to create a new commit where we add a new file within the ‘.github/workflows’ folder. But to do that, we first need to encrypt the content using Base64. If your goal is to only scan the code, without further feedback, then this should be your last step. You could directly check the result on your GitHub Repository page.
  Side Note: You might want to edit the YAML file that is used within the ‘content’ parameter, as it will be different for each programming language, in this case, it would only work for codes written in C#.

# URL for creating a commit
$url = "https://api.github.com/repos/$(GithubUsername)/$(Build.Repository.Name)/contents/.github/workflows/codeql.yml"

# Create header with PAT
$token = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes("$(GithubPAT)"))
$header = @{authorization = "Basic $token"}

$projectJSON = @{message = "feat: add workflow from devops"
               content = "bmFtZTogIkNvZGVRTCIKCm9uOiBbcHVzaF0KCmpvYnM6CiAgYW5hbHl6ZToKICAgIG5hbWU6IEFuYWx5emUKICAgIHJ1bnMtb246IHVidW50dS1sYXRlc3QKICAgIHBlcm1pc3Npb25zOgogICAgICBhY3Rpb25zOiByZWFkCiAgICAgIGNvbnRlbnRzOiByZWFkCiAgICAgIHNlY3VyaXR5LWV2ZW50czogd3JpdGUKCiAgICBzdHJhdGVneToKICAgICAgZmFpbC1mYXN0OiBmYWxzZQogICAgICBtYXRyaXg6CiAgICAgICAgbGFuZ3VhZ2U6IFsgJ2NzaGFycCcgXQoKICAgIHN0ZXBzOgogICAgLSBuYW1lOiBDaGVja291dCByZXBvc2l0b3J5CiAgICAgIHVzZXM6IGFjdGlvbnMvY2hlY2tvdXRAdjMKCiAgICAtIG5hbWU6IEluaXRpYWxpemUgQ29kZVFMCiAgICAgIHVzZXM6IGdpdGh1Yi9jb2RlcWwtYWN0aW9uL2luaXRAdjIKICAgICAgd2l0aDoKICAgICAgICBsYW5ndWFnZXM6ICR7eyBtYXRyaXgubGFuZ3VhZ2UgfX0KICAgICAgICBxdWVyaWVzOiBzZWN1cml0eS1leHRlbmRlZCxzZWN1cml0eS1hbmQtcXVhbGl0eQoKICAgIC0gbmFtZTogQXV0b2J1aWxkCiAgICAgIHVzZXM6IGdpdGh1Yi9jb2RlcWwtYWN0aW9uL2F1dG9idWlsZEB2MgoKICAgIC0gbmFtZTogUGVyZm9ybSBDb2RlUUwgQW5hbHlzaXMKICAgICAgdXNlczogZ2l0aHViL2NvZGVxbC1hY3Rpb24vYW5hbHl6ZUB2MgogICAgICB3aXRoOgogICAgICAgIGNhdGVnb3J5OiAiL2xhbmd1YWdlOiR7e21hdHJpeC5sYW5ndWFnZX19Ig=="
            } | ConvertTo-Json

Invoke-RestMethod -Uri $url -Method Put -ContentType "application/json" -Headers $header -Body ($projectJSON )

• [Wait for CodeQL to complete] (Optional)
  If you wish to get feedback from Github back to Azure DevOps after the CodeQL script is sent and run in the Github Repository, we must first wait for it to complete to get the results. For that, we would be hitting GitHub API again, in this case, every 30 seconds until the scanning status is completed.

# URL for checking Github Actions status
$url = "https://api.github.com/repos/$(GithubUsername)/$(Build.Repository.Name)/actions/runs?per_page=1"

# Create header with PAT
$token = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes("$(GithubPAT)"))
$header = @{authorization = "Basic $token"}

# Wait for Github Actions
do {
    Start-Sleep -Seconds 30
    $Response = Invoke-RestMethod -Uri $url -Method Get -ContentType "application/json" -Headers $header
    $Status = $Response.workflow_runs[0].status
    Write-Host "Status: " + $Status
} while ($Status -ne "completed")

# Testing variables
$Response.workflow_runs[0].conclusion
Write-Host $Response.workflow_runs[0].head_sha
Write-Host $Response.workflow_runs[0].created_at

# Fail task if CodeQL returns failure
if($Response.workflow_runs[0].conclusion -eq "failure") {
    echo "##vso[task.logissue type=error]Github Code Scanning Failed"
    exit 1
}

Write-Host "Github Code Scanning Completed"

• [Check Code Scanning result] (Optional)
  For the final step, we’re going to hit GitHub’s API to get the list of vulnerabilities within our code. In this case, the script would throw an error if the code got at least 1 critical or 1 high vulnerability. You could customize the threshold for an acceptable code in the if statement of this script.

# URL for checking Code Scanning Result
$url = "https://api.github.com/repos/$(GithubUsername)/$(Build.Repository.Name)/code-scanning/alerts"

# Create header with PAT
$token = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes("$(GithubPAT)"))
$header = @{authorization = "Basic $token"}

$critUrl = $url + '?severity=critical'
$critResp = Invoke-RestMethod -Uri $critUrl -Method Get -ContentType "application/json" -Headers $header
$crit = $critResp.Count

$highUrl = $url + '?severity=high'
$highResp = Invoke-RestMethod -Uri $highUrl -Method Get -ContentType "application/json" -Headers $header
$high = $highResp.Count

# Throw error if Code Scan found vulnerabilities
if (($crit -gt 0) -or ($high -gt 0)) {
    Write-Host "##vso[task.logissue type=error]Code Scanning Found Vulnerabilities:" $crit "Critical," $high "High"
    exit 1
}

Now that we know the scripts that are required for accomplishing this task, we could create a new task group that contains these steps that makes it easier if we want to do it for multiple pipelines (Ref: Task groups in Azure Pipelines and TFS (classic)). Utilizing the feature that allows us to get the code scanning result, we could further add custom tasks such as sending email notifications to the user that created the build, among other things to further streamline our workflow.

References:

• Integrate security into your developer workflow with GitHub Advanced Security for Azure DevOps
• Azure DevOps supports the “CodeQL” service for Repo’s SAST scan.
• Managing your personal access tokens
• Task groups in Azure Pipelines and TFS (classic)