Faster Azure DevOps pipeline development loop

Have you ever tried creating a brand-new Azure DevOps pipeline and executing it just to realize that you had an error on your script that needed you to go back and forth to be corrected just to get it right?

I also have suffered from this throughout my career when creating not-so-straightforward pipelines, like when needing to modify a JSON object before deployment of an Azure Data Factory or when required to use a mix of Azure tools, such as Azure CLI and rest API to get a certain job done.
This article will cover several ways to improve the inner cycle development and reduce this back and forth when developing a pipeline in Azure DevOps. These ideas revolve around the tooling, the out-of-the-box features, and the scripting.

Note: Some ideas presented in this article could also be used in other CI tools.

© pixabay

Syntax highlighting/autocompletion/error detection

Several tools can help write a better pipeline code with features like code highlighting, autocompletion, and syntax error detection. Here are some examples of tools with these features:

• Visual Studio Code extension Azure pipelines provide syntax highlighting, autocompletion, and formatting on the pipeline YAML. These features will help you get the right pipeline syntax and avoid some of the need to check the documentation.

• Visual Studio Code extension ShellCheck to help write bash/sh scripts. This extension highlights some syntax issues and frequent errors.
• Visual Studio Code extension PowerShell to help write PowerShell scripts. This extension provides syntax highlighting, intellisense, and built-in code snippets to be used with a simple shortcut and frequent error detection.
• The built-in validate button on Azure DevOps will ensure the pipeline is valid. Be sure to use it whenever editing directly on the portal

Enable verbose logging on pipeline runs

Turning on the verbose logging on the Azure DevOps pipeline will help troubleshoot problems on the pipeline. This can be done either by setting the option “Enable system diagnostics” to enable it once or setting the variable System.Debug to true.

variables:
  - name: System.Debug
    value: true

Writing and testing scripts outside the pipeline

Creating a script file outside the pipeline, allows it to be easily tested and debugged outside of the pipeline execution context. Any dependency related to the CI environment, such as environment variables could be set before executing the scripts.

In Azure DevOps, this could be achieved by creating simple pre-run scripts in PowerShell like:

$env:BUILD_ARTIFACTSTAGINGDIRECTORY=".\source-code" #to represent the agent variable Build.ArtifactStagingDirectory

or in bash like this:

BUILD_ARTIFACTSTAGINGDIRECTORY="./source-code" #to represent the agent variable Build.ArtifactStagingDirectory

PowerShell and bash scripts can be easily debugged using VSCode described in this article https://devblogs.microsoft.com/scripting/debugging-powershell-script-in-visual-studio-code-part-1/ for PowerShell and bash using https://marketplace.visualstudio.com/items?itemName=rogalmic.bash-debug

ChatGPT

ChatGPT can be used for several tasks such as: drafting simple pipelines or parts of them, understanding pipeline errors, and creating PowerShell and bash scripts.

When asking for the simple pipeline below with ChatGPT 3.5:

Create an Azure pipeline that runs only on the main branch and runs on ubuntu for a Web API
on dotnet core 6 that build a container image from dockerFile on "./WebApplication1/Dockerfile"
named webappdemo and push it to dockerHub.
 
This pipeline should use the containerRegistry property on task  Docker@2 with value "dockerHub".

It provided the following result:

trigger:
- main
 
pr:
- main
 
pool:
  vmImage: 'ubuntu-latest'
 
jobs:
- job: BuildAndPushDockerImage
  displayName: 'Build and Push Docker Image'
  steps:
  - task: UseDotNet@2
    displayName: 'Install .NET Core SDK 6.x'
    inputs:
      packageType: 'sdk'
      version: '6.x'
      installationPath: $(Agent.ToolsDirectory)/dotnet
 
  - script: dotnet --version
    displayName: 'Check .NET Core Version'
 
  - checkout: self
    persistCredentials: true
 
  - task: Docker@2
    displayName: 'Build and Push Docker Image'
    inputs:
      containerRegistry: 'dockerHub'
      repository: '<your-dockerhub-username>/webappdemo'  # Replace with your Docker Hub username
      command: 'buildAndPush'
      Dockerfile: './WebApplication1/WebApplication1/Dockerfile'
      tags: |
        $(Build.BuildId)

The result comes out very fast and can provide a basis for beginners, despite not being perfect.

When trying more advanced scenarios with ChatGPT 3.5 I found that it couldn’t very well create ready-to-use pipelines. Be aware when using ChatGPT to not end up losing more time than gaining.

Another advice when using this is to use a newer version, such as version 4, which is more up-to-date and has better generative text capabilities.

Use and abuse of templates

Repeating the same pipeline task, job or stage is a very tedious and error-prone task. Try using templates with parameters to maximize reuse across pipelines.

Here’s an example of a template definition to build and test a dotnet app:

Dotnet Build Template

parameters:
- name: buildProjects
  default: ''
 
stages:
- stage: Build
  displayName: Build
  jobs:
  - job: Build
    strategy:
      matrix:
        Release:
          _buildConfig: 'Release'
        Debug:
          _buildConfig: 'Debug'
    pool:
      vmImage: 'windows-latest'
    steps:
      - task: UseDotNet@2
        displayName: 'Use .NET Core sdk'
        inputs:
          packageType: sdk
          version: 7.0.x
          installationPath: $(Agent.ToolsDirectory)/dotnet
 
      - task: DotNetCoreCLI@2
        displayName: Restore
        inputs:
          command: 'build'
          projects: ${{ parameters.buildProjects}}
          arguments: '--configuration $(_buildConfig) /t:restore'
       
      - task: DotNetCoreCLI@2
        displayName: Build $(_buildConfig)
        inputs:
          command: 'build'
          projects: ${{ parameters.buildProjects}}
          arguments: '--no-restore --configuration $(_buildConfig)'
      - publish: $(System.DefaultWorkingDirectory)/bin
        artifact: $(_buildConfig)Build

and how to use this template:

trigger: none
 
pr:
  branches:
    include:
    - main
    - release/*
    - feature/*
 
variables:
  _libraryProjects: |
    src/*.csproj
 
stages:
- template: templates/BuildStage.yml
  parameters:
    buildProjects: $(_libraryProjects)

Conclusion

This article presented some easy techniques to save time when creating an Azure DevOps pipeline, but there are still many more that can be used and ultimately the best way to save time is experience from practicing. I hope this will help you dear reader 😀