Different facets of YAML based Azure DevOps Pipeline

This article provides information about some important features that we can use in Azure pipeline. Azure pipelines have tons of task plugins and enough features to handle most common CICD flows. Azure YAML syntax for Azure pipeline is proprietary and can not be used with other CICD platforms.

Introduction:

Azure DevOps is a DevOps server offered by Microsoft which helps you to simplify DevOps processes. Azure DevOps provides Version Control (Azure Repo), project management (Azure Boards), testing (Test Plans) and CI/CD service (Azure Pipelines).

Azure DevOps allows you to create a pipeline in two different formats: classic editor which is GUI based format for which no prior knowledge is required and other is YAML based format, where basic understanding of YAML is required.

Azure YAML allows you to define your CI/CD strategy as a code and the pipeline file is then committed to repo. Due to this you can take advantage of pull request, history and code review just like other application code. The YAML pipeline file is version control enabled so you can rollback to any previous working version.

In classic editor, you can define the pipeline in the Azure DevOps with graphical user interface. Classic editor pipeline can not be a part of version control so changes are not tracked. Using classic editor, you can create build and release pipelines in DevOps portal, in that your build pipeline will create artifacts which can be used for code deployment in environments like staging and production.

Below points are covered in the article for the YAML pipeline:

1. Prerequisites
2. Predefined Variables
3. Agent Node Pool
4. Run Time Parameters
5. Trigger one Pipeline from another
6. Publish and Consume Artifacts in Pipeline.
7. Azure DevOps Secure Library
8. Conclusion
9. References

1. Prerequisites:

• Azure DevOps account
• Basic knowledge about YAML
• Find more details here

2. Predefined Variables:

In Azure DevOps by default many variables already exist. Such variables are called pre-defined or system variables. Predefined variables are automatically set by system and are read-only, they can represent simple strings and numbers. Below are some of the commonly used pre-defined variables.

2.1 ${Agent.BuildDirectory} / ${Pipeline.Workspace}: This variable stores local path on agent for running build, both variables have the same values.

Example: /home/pipelineagent/_work/1

2.2 ${Agent.TempDirectory}: A temporary folder that is cleaned after each pipeline job. You can use this variable to store secret files during the pipeline and clean it up after that.

2.3 ${Build.BuildId} / ${Build.BuildNumber}: The ID for the completed build

2.4 ${System.DefaultWorkingDirectory}: The local path on the agent where your source code files are downloaded.

3. Agent Node Pool:

To run, build and deploy a pipeline for your application you need at least one agent. When you configure your pipeline, you have two agent options. One is Azure managed agent and another is self-hosted agent. Agent can be a virtual machine or a container which acts as a compute engine. Agent operating system can be Windows, Ubuntu, Mac OS.

The important point to consider here is the cost. With Self-hosted agent you can use the Azure pipeline for free. Here you are able to provide compute resources and use Azure devops pipelines as an orchestrator. Self hosted agent gives you the privilege to install software or libraries required for your build and deploy pipeline.

Whereas, if you have not exceeded the Azure pipeline’s quota, you can use the Microsoft-hosted agent without any cost.

• The Azure pipeline’s quota includes: 1 Microsoft hosted agent with 1800 minutes per month for CI/CD and 1 self-hosted job with unlimited minutes per month.

3.1 How to use Microsoft-hosted agent?
To use Microsoft-hosted agent in your Azure pipeline, mention the operating system name as shown in below YAML format:

pool:
  vmImage: ubuntu-20.04

3.2 How to use Self-hosted agent?

1. Go to Organization Settings.
2. Click on Agent Pools.
3. Click on Add Pool.

4. Fill the details as shown in the picture below & on next screen click on New Agent.

5. Open New Tab, To configure a new agent we will need PAT (Personal Access Token), refer below screenshot to create PAT:

Azure DevOps PAT

Azure DevOps Create PAT

Note: Don’t forget to copy the token as soon as you generate as you will not be able to see it again.

6. Go back to the Agent Pools page and download the agent and install it by following instructions provided by Microsoft. While running it you have to provide PAT and other details so that it can be configured.

Once your agent is configured you can see your agent in Azure DevOps portal.

4. Run Time Parameters:

Previously, the parameters were passed using the variables section defined in the pipeline. The main issue with the earlier way was all variables were string, so there was no validation to check if the correct value had been passed or not.

For example:

Now we have an option to define runtime parameters, including parameters key that takes the boolean parameter value. Parameters should be set at the beginning of a YAML. In the example below, the trigger is set to none so that you can choose the value of installNewrelic, by default, it is configured as false.

Azure DevOps YAML Pipeline

When you run the pipeline manually you will be presented with a user interface (shown below) that allows the user to select this runtime parameter before running the pipeline. If you don’t make a selection, the default option false is used and the ‘Install NewRelic’ task will be skipped.

5. Trigger one pipeline from another:

In some cases we have pipeline dependency on each other, in such cases it can be helpful to trigger a pipeline from another pipeline. All you need to do is add below code in your YAML pipeline section.

# this is being defined in org.devops.demo – QA pipeline
trigger: none
resources:
  pipelines:
  - pipeline: DevBuild
    source: org.devops.demo - DEV
    branch: develop
    trigger:
      branches:
      - develop

By adding trigger: none QA pipeline will not trigger on commit & only triggers when DEV pipeline finishes it’s job.

6. Publish and Consume Artifacts in Pipelines:

Azure DevOps pipeline artifacts allow you to share files between stages in a pipeline or between different pipelines. Artifacts are output of the build process that needs to be deployed or consumed by another job.

6.1 To publish or upload artifacts in a pipeline:

steps:
- publish: $(System.DefaultWorkingDirectory)/app
  artifact: myapp

Explanation:

- To publish an artifact path of the file or folder is required.

- Artifact name is an option, but it is good to have a name so that you can use them anywhere in the pipeline.

6.2 Download Artifacts in same the pipeline.

steps:
- download: current
  artifact: myapp

Explanation:
- current means current run of the pipeline.

6.3 Download Artifacts in another pipeline:

Use the below task in the YAML pipeline to consume artifacts from another pipeline.

# this is being defined in org.devops.demo – QA pipeline
steps:
- task: DownloadPipelineArtifact@2
  inputs:
    source: 'specific'
    project: 'DevOps Demo'
    pipeline: 786 # ID of the pipeline from which you are downloading artifacts
    runVersion: ‘latestFromBranch’
    preferTriggeringPipeline: true
    runBranch: develop   # Branch of the artifacts pipeline
    artifact: 'myapp'    # Artifact Name
    path: '$(Pipeline.Workspace)/myapp

7. Azure DevOps Secure Library:

Sometimes we have credentials which you can’t commit in repo, but we want to use that file and deploy on the application server in a secure way. In this case we can use Secure Library which allows us to store credentials file, download and upload on server through pipeline.

To upload file in Secure Library:

Go to Azure DevOps → Click on Pipelines → Library → Click on Secure Files → Click on “+ Secure file”

Example: Here I have uploaded secret.php

To download file through pipeline from Secure Library:

- task: DownloadSecureFile@1
  name: mySecretFile # Any name which you can refer further in the pipeline.
  displayName: 'Download Secret File'
  inputs:
    secureFile: 'secret.php'
- script: |
    cp $(mySecretFile.secureFilePath) $(Pipeline.Workspace)/config

Here,
- Above code will download secret.php at $(Pipeline.Workspace)/config directory.

8. Conclusion:

To summarize, this article discusses various features in Azure DevOps pipelines which provides control on CI/CD and the best thing is that everything is done using code. Declarative YAML pipeline adds cherry on top due to its easy to use syntax and it has a lot of amazing features to be explored (example — Caching feature to reduce pipeline execution time, custom conditions etc.). I hope this will help readers to adopt and use YAML based Azure DevOps pipeline without any hassle.

9. References:

• For more details about Azure Pipelines, please visit the official documentation.
• Environment is also one of the feature of Azure DevOps Pipeline, Click here to read more.
• Read More blogs http://www.anup.co.in

Hope you found this article useful!