Using templates in Azure DevOps Pipelines: What, Why and How!

Are you starting to use Azure DevOps Pipelines to build and deploy services, and need some clarity around how to structure your pipelines? This article explains some basic concepts regarding templates, how to use them to create generic and scalable pipelines, and also provides a functional GitHub repo to try them out. I will cover an approach to create pipelines for an application containing multiple repositories using templates.

Why pipelines using templates?
The typical approach to create pipelines is to create a YAML file and add all necessary stages and tasks in a single YAML file. The disadvantage of this approach is that it’s not scalable. It may be good for an application with only one repo, but if we add more repositories then the tasks in the original YAML need to be duplicated in the new pipelines. This violates the DRY principle.

Here is an example of a pipeline without using templates:

addition-service-without-template.yml

As can be seen above that all the tasks are in one YAML file which is not a clean approach to create pipelines.

To avoid this, we can create pipelines using templates where each template is created generically and dedicated to a particular task which then can be reused in multiple repositories. More information regarding templates can be found at Microsoft documentation.

GitHub Sample Application: Let us take an example of an application that contains two microservices.
1. addition-service: This is a python flask application that exposes the rest endpoint to calculate the addition of two numbers.
2. multiplication-service: This is a java spring-boot application that exposes the rest endpoint to calculate the multiplication of two numbers.
To deploy these services we can create Docker images, which then can be deployed to AKS(Azure Kubernetes Service).

GitHub link to this project can be found here https://github.com/anandchugh/pipelines-using-templates
To use the repo, download it and import the pipelines in Azure DevOps.

Identify common tasks during build and deployment:
Let us first identify some basic tasks which are required to build and deploy these microservices.

1. Build Related Tasks
   - Build source code
   - Run Unit tests
   - Publish Unit tests
   - Enable Code coverage
   - Publish Code Coverage
   - Quality check for code coverage.
2. Docker Related Tasks
   - Build a docker image
   - Tag docker image
   - Push the docker image to the container registry.
3. Deployment Related Tasks
   - Download docker image
   - Deploy docker image to Kubernetes service using kubectl or as a helm chart.
4. Credential scan of source repo for any secrets or passwords.

Creating pipelines using templates for the above tasks
Build related tasks require tools for building, which are specific to a particular language, hence for our current application we can create two build YAML files one for each python and one for java.

Here are the two build pipelines:

addition service pipeline

multiplication service pipeline

As can be seen above, both individual pipelines are quite small and most of their tasks are encapsulated in templates.

In GitHub repo, the following are generic templates created, which are part of a common folder:

ci-build-java-template.yml

ci-build-python-template.yml

ci-coverage-check-task-template.yml

ci-docker-template.yml

addition-service is using these templates:
1. ci-cd-vars-template.yml: This contains variables used in the pipeline.
2. ci-build-python-template.yml: This encapsulates various tasks like:
- build
- run and publish the unit test
- run and publish code coverage
- code coverage checks
3. ci-docker-template.yml: This template currently encapsulates only the docker build step, but we can add other steps like docker image tagging and docker image push in the same template.
4. Finally, as we have added build and docker stage we can add a deployment stage template that can use kubectl or helm to deploy the service in Kubernetes service.

Similarly, here is the list of templates used by multiplication-service pipeline:
1. ci-cd-vars-template.yml: This is shared between all pipelines.
2. ci-build-java-template.yml: This is specific to java repo which does build, unit tests, and code coverage tasks.
3. ci-docker-template.yml: This is also shared between all pipelines.
4. Once we add a deployment template, we can reuse it here also.

Hence, we can see here that templates which are reused and shared between both the pipelines are:
1. ci-cd-vars-template.yml
2. ci-coverage-check-task-template.yml
3. ci-docker-template.yml
We can also use parameters to pass any dynamic values to the templates as it is done with the ci-cd-vars-template.yml template in both the pipelines.

If we add any new service to the project, then creating a pipeline for that is quite straightforward as we can just reuse the existing templates. For example, if we create a new service subtraction-service in python we can create a pipeline like this.

subtraction-service pipeline

The only difference between addition-service and the subtraction-service pipeline is:
- project name
- source directory of the project
and all other templates are reused.

Conclusion:
Once we have generic templates for various tasks related to build and deployment, we can reuse those tested and verified templates in other pipelines and once we do that, pipelines can be created without much effort. This will definitely reduce the effort to create pipelines from hours to minutes.