Automation, Documentation and best practices for your DevOps Yaml Pipelines using Githooks and powershell

In my previews post, I wrote about the advantages of using a separate DevOps Repository to keep all of the pipeline and template YAML files separate from the code repository.

Yaml Pipelines a multi-repository approach

In this post, I want to share the best practices you can use to organise all of our YAML based pipeline and template files that you have in the DevOps repo and how to Document the relations between them when you follow the multi-repository approach.

The end goal is that we can automatically generate the repository documentation via a script.

To get there, pipeline YAML files are grouped by Projects into folders

.\<ProjName>-API-Pipelines

.\<ProjName>-Core-Pipelines

Within the Folder the Pipeline files are grouped by the Pipeline type, for instance, a Folder named CI for Continues Integration or Release for a Release Pipeline, each folder has the YAML file that holds the tasks of the pipeline as well as template YAML files that make common tasks reusable between the pipelines.

The YAML Pipeline files have the file extension *.pipeline.yml and the templates *.template.yml to make it easier to differentiate between them.

The Templates will be stored in the lowest folder with which they are shared with so for example the template NodeAndGulpTasks.template.yml is used by the CI and Release Pipeline of the core project so it is within the SharedResources folder of the Core project folder. The GenerateVersionString.template.yml template is used by the API and Core Project so it will be in the folder called .\SharedResources directly on the root.

.\PN-API-Pipelines\CI
.\PN-API-Pipelines\CI\PN-API_CI.pipeline.yml
.\PN-API-Pipelines\Release\PN-API_Release.pipeline.yml
.\PN-Core-Pipelines\CI\PN-Core_CI.pipeline.yml
.\PN-Core-Pipelines\Release\PN-Core_Release.pipeline.yml
.\PN-Core-Pipelines\SharedResources\NodeAndGulpTasks.template.yml
.\SharedResources\GenerateVersionString.template.yml
.\SharedResoures

Each Pipeline and template file contains xml tags with useful Information like:

• Details on the purpose of the File <summary>
• Parameters used in case of Templates <param>
• and other useful information <remarks>

For example:

# <summary>
# This is the Build and Release Pipeline file for the <ProjName> Service apps and functions
# It will deploy directly to the relevant Azure resource
# Each environment has its on stage
# </summary>
#<remarks>
# This Pipeline file uses Variables and parameters,
# the parameters are used because they allow for objects with multiple properties to be defined
# this cannot be done using variables
#
# Each environment has a variable group in the DevOps library, see https://dev.azure.com/<Organisation>/Project/_library?itemType=VariableGroups
# the variable groups are mapped to the Stage(Dev, QA, Prod) to get the corresponding settings for each environment
#</remarks>
trigger: none #Removes CI Trigger on the DevOps Repo
#Get Resources
resources:
 repositories:
  - repository: <ProjName>-Services # code repository
    name: <ProjName>/<ProjName>-Services
    type: git
    trigger: # Release trigger for this repository
    - ‘master’
........

Those XML tags and the file name standardisation come in handy now because we can use a script I put together to generate the Documentation for the DevOps Repo automatically and publish it to a DevOps Wiki.

You can download the script from my Github page

/ReGenerateReadMe.ps1

It will go throe all the .pipeline.yml files, get the XML tag contents and a list of all the templates the pipeline uses.

In the same GitHub repo as the script, I uploaded some pipeline files with XML tags and the Readme.md file that got generated from the script. https://github.com/qBasicBoy/SampleDevOpsRepo to demo this in action.

The Readme.md file’s first section is the content of the __Readme.md file, this is so that we can start with some general Information. The Second part is a list of all the Template and Pipeline files with the information from the XML Tags that the PowerShell script generated.

The pipeline block lists all the *.pipeline.yml files with the Info from the XML tags and the template block lists all the *.template.yml files with the info from those tags.

For example, the SetBuildName.Template.yml has the following tags

The Readme File generated by the script has the summary blog and a remarks block, it lists all the parameters the template accepts and what I find very useful, is it shows you a list of all the pipelines that use the template.

Lastly, we can add a githook by creating a file called “pre-commit” in the folder \.git\hooks and adding the following content:

#!/bin/sh 
commit”powershell -file ReGenerateReadMe.ps1 args

This will execute the script every time you create a new git commit.

In Azure DevOps you also have the option to generate a wiki page from code and use the readme.md file that the script generated.