YAML Pipelines; a multi-repository approach, separate your YAML from your code

Separating YAML pipelines from your code has 3 big advantages;

First, if you keep your DevOps Pipeline YAML files in a separate Repository you don’t have to worry about Pipeline changes messing with your Git-flow main, Development and Release branch strategy. You simply run your pipeline from your main branch in the DevOps Repository and then check out whichever branch you need, be it the Release or Development branch you want to deploy from your code Repository.

Second, you can manage the work from each team in separate Repositories like a Quality Assurance Repository to host the automated tests.

Thirdly, If you keep all your DevOps resources like YAML Pipeline, YAML Template and Scripts in a dedicated Repository you can share them across Pipelines that build different code git Repositories, for example, I commonly see that the Azure Function code is in a separate git Repository than the MVC Web Project but you may want to share the template that versions the code across all your code build Pipelines.

For me, working for a company where the development of a cms based website is outsourced to an offshore team with a separate in-house QA and DevOps team, my goal was to facilitate a pipeline structure that lets each team, work as independently as possible and keep Repositories and Pull Requests separate from each other.

We identified the need for 3 Repositories one for the Outsourced Dev team working on the Website, one for the QA team writing the automated tests and lastly the DevOps team to hold the PS Scripts and the YAML files for the pipelines them self.

This way each team can have access to their respective Repository, their own branches and ways of raising and reviewing Pull requests.

To get there first we created a new Repository called DevOps, then a new pipeline which we pointed to the DevOps Repository to host the YAML files.

Next we update the YAML file, in our case, CMS-Release.pipeline.yml to use our Internal Agent and add 2 Repositories as resources, the WebsiteCms and the QA-Automation Repository.

trigger: none #Removes CI Trigger on the DevOps Repo
pool:
 name: Internal Agents #use internal build agent
 demands:
  - npm
  - node.js
resources:
  repositories:
- repository: WebsiteCms # code repository
    name: Website/Website.Cms
    type: git
    ref: develop2022
  - repository: QA-Automation # code repository
    name: Website/QA-Automation
    type: git
    ref: main

Note how we didn't specify the DevOps Repository as a resource, that’s done automatically since we pointed the pipeline to the CMS-Release.pipeline.yml file in the DevOps Repository when we created it.

Then in the subsequent stages, we check out the resources (Repos), that’s the actual Code repo (WebsiteCms) and the DevOps Repository in which we have all the YAML template files and PowerShell scripts our pipeline needs.

stages:
- stage: Build
  displayName: Build Website CMS
  jobs:
  - job: Main
    displayName: Run Build Tasks
    steps:
    - checkout: WebsiteCms
       path: s/WebsiteCMS
    - checkout: Self
       path: s/DevOps

We checked out the WebsiteCms to the path s/WebsiteCMS and we can check out the DevOps Repository even though we didn't specify it as a resource earlier by just specifying “self”.

We can then get to the resources like YAML templates and scripts etc.. within the DevOps Repository that has been downloaded to the “s/DevOps” path we specified.

- template: ‘s/DevOps/SharedResources/BuildCMS.Template.yml’
  parameters:
   FormatedVersion: $(FormatedVersion)
   SourcesDirectory: s/WebsiteCMS

After building and Deploying the website using the WebsiteCms and DevOps Repository I can run the automated tests against the deployed site using the QA-Automation Repository resources.

- stage: AutomatedTests
  displayName: UAT Automated Tests
  dependsOn:
   - UAT
   - Build
   - WaitForUATSite
   jobs:
    - job: RunTests
      timeoutInMinutes: 360
      displayName: Automated tests Job
      variables:
      - group: URLs
      steps:
      - checkout: QA-Automation
        path: s/QA-Automation
        - template: ‘s/DevOps/SharedResources/CypressTests.Template.yml’
          parameters:
           emailList: $(Build.RequestedForEmail)

If we want to use a different branch on any of the Repositories, we can update the branch before kicking off the Pipeline in the resource tab.

The described approach allowed us to create a multi-stage, multi-deployment and test pipeline process looking like this.

In my next post I go over some standardisation I found useful to organise the DevOps Repository.

automation documentation and best practices for your devops YAML-pipelines using githooks and Powershell