Audit Azure DevOps configuration with Sarif scan reports from the Pipeline

4 min readSep 26, 2023

In my previous article here, I walked through the steps of auditing your Azure DevOps configuration with the new PSRule.Rules.AzureDevOps module for PSRule. The module is in active development and in the new 0.0.12 release it includes a sample YAML pipeline definition for auditing your own environment with a nice report on the Scans tab in the Azure DevOps portal.

Sarif output from PSRule.Rules.AzureDevOps scan

Getting started with you own pipeline

To get started with your own pipeline, a sample `azure-pipelines.yml` is available in the project’s GitHub repository. The file can be downloaded here. Download the file and then go to your own Azure DevOps Project at https://dev.azure.com/<your-organization>/<your-project>.

Add the downloaded file to either a new repository or use your project’s existing git repository and add the file there.

Add the YAML to your Azure DevOps repository

To start with a new repository from scratch:

In the Azure DevOps portal click `Repos` in the left menu.
In the top ribbon, click the default repository’s name to expand the menu and then click `New repository`

Enter `psrule-scan-ado` as the name for your new repository, make sure `Add a README` is checked and click `Create`.
Click the `More actions` menu button on the top right of the repository pane and click `New` -> `Folder`
In the New folder name field enter `pipelines` as folder name and `azure-pipelines.yml` as the New file name.

Create a new file and folder in Azure DevOps repository

Azure DevOps now opens an editor, paste the contents of the `azure-pipelines.yml` file we downloaded earlier in the editor but don’t save/commit and close yet.
In the editor, go to the value for `devops_organization` on line 11 and change it to the name of your Azure DevOps organization.
Do the same for the name of your Azure DevOps project on line 14 with the value of `devops_project`.
The schedule is set on line 17 and the current settings run the audit every sunday.
Commit your changes and confirm the commit with a message.

Create a Personal Access Token

The PSRule.Rules.AzureDevOps module needs access to the Azure DevOps REST API to scan your project. To authenticate it requires Personal Access Token (PAT) with full access permissions. To create the PAT:

Click `User settings` on the top right of the Azure DevOps portals and then click `Personal Access Tokens`.

Click `New Token`.
Enter a name for the token and set the Scope to `Full Access`.
Set the expiration for the token as desired and click `Create`.
Copy the created token to your favorite password manager or temporary notepad.

Create variable group with the PAT

To keep this step-by-step guide concise and simple, check the Quickstart how to create a Key Vault on how to create a key vault to securely store the ADO pat. Create the Key Vault as described in the guide and add the generated PAT from the previous section as a secret named `ADOPAT`. Once you have the `ADOPAT` secret created follow the steps below.

Return to your project in the Azure DevOps portal.
Click `Pipelines` in the left menu.
In the `Pipelines` menu, click `Library`.
In the `Library` pane top ribbon, click `+ Variable group` to add a variable group.
As Variable group name enter: `ado-psrule-run`.
Select `Link secrets from an Azure key vault as variables`.
Select your subscription and key vault name in the selectors.
Then click `+ Add` to add the `ADOPAT` secret as a variable to the new group.
Click `Save` in the top ribbon.

Add and run the pipeline

Last thing we need to setup is the actual pipeline itself in Azure DevOps. The pipeline is defined by the YAML definition we created in the earlier steps and uses the Personal Access Token we stored in Key Vault.

In the left menu click `Pipelines` and again `Pipelines` below the main Pipelines Category.
On the top right, click `New pipeline`.
Select `Azure Repos Git (YAML)`.
Select the `psrule-scan-ado` repository.
Select `Existing Azure Pipelines YAML file`.
Select `main` as branch.
Select `/pipelines/azure-pipelines.yml` as Path.
Click `Continue`.
Click `Run`.

Your pipeline will now run and scan your Azure DevOps environment. The run may very well show up as a failure depending on the results of the scan. If no issues are found the pipeline result will be a success. If any issues are found, the PSRule task will show the test as a fail and thus fail the pipeline. After the run finishes you can view and filter the results in the `Scans` tab of the run result page.