Reusing Tasks in Azure DevOps Pipelines

As we are aware, DevOps has changed the way products are built and deployed over the years and responsibility of build and deployment management has shifted from traditional build engineers to automated processes. This means the chances of human induced errors have drastically reduced.

We at Kongsberg Digital, are constantly looking for opportunities to better the software quality and to reduce manual efforts during deployment. Our products are rapidly adapting to continuous deployment model, which means development cycles are short and every piece of code is instantly deployed to test, staging and production environments. Automation plays a big role is this process and we heavily rely on Azure DevOps Pipelines tool for this. With so many different CI/CD pipelines performing a variety of functions including integration, testing and deployment, it is inevitable to have tasks that are repetitive.

Task Groups

A good continuous integration process will, at the bare minimum, have two pipelines. The first one tests the code changes and the second publishes artifacts after the changes are integrated.

If you take the example of the above shown pipelines, the first four tasks are common to both definitions. In case there is a need for modifying one of the tasks (or remove/add a common task), the change has to be separately applied to both definitions. Now think about it at a larger scale, where there are many pipelines and you have to modify each one’s definition.

This is where Azure Pipeline’s Task Groups come in handy. Task Group, as the name suggests, groups a common set of tasks into one single task. Multiple pipelines can now use this new task instead of repeating the steps all over again. Creating task groups is very easy. Just select the tasks that you want to group and choose “Create task group” from the context menu. A popup appears, enter name and description and select category under which this new task should appear and click on the “Create” button. It is also possible to create a task group that encapsulates other task groups.

At the time of writing this article, Task Groups are not supported in YAML Pipelines. Templates are a good alternative to that.

Once created, the task group is now available from “Add Task” panel.

If you look at our example pipelines, the common repeating tasks are now grouped and only the task group has to be updated in case of any future changes.

Task Group Versioning

A major problem that we have to deal with, is maintaining a task group. Making small changes is fine, but how do we handle major changes like introducing a new mandatory parameter for the build/release pipelines to set or changing the version of .NET Core SDK that one of your tasks inside the group is using. You could soon end up with many broken builds with one such update.

Azure Pipelines has a nice “blink-and-you-miss-it” version control feature that is specifically used in scenarios like this. It uses semantic versioning. For minor changes, you can follow the normal path and click on Save button (shown in the below given figure). Internally, this bumps the minor version of the task group.

The pipelines that are using this task group will automatically receive latest changes. This happens because Azure Pipelines uses most recent minor version of the selected major version. This is in-built and there is no way of selecting exact versions (Although, I believe, this feature can be of use is some cases).

Bumping major version of task group

It is important to consider impact of your changes. If they are incompatible with existing pipelines, then you have to bump major version of the task group. This happens in stages and is explained below:

Stage 1 — Draft Version

After you are done with the changes, click on Save as draft instead of Save button.

This creates a separate draft version of your task group. You will notice “-test” being appended to the version number. Draft version of the task group is separately available in the “Add tasks” panel. This allows you to test your changes in a test pipeline. Other build/release pipeline can continue to use the original base version.

At this stage, you are allowed to make changes to the base version of the task group. However, these changes will not reflect in the draft version.

Stage 2 — Publish

Once you are satisfied, it is time to publish the changed task group. And this you do by clicking on the Publish draft button. You have the option to either publish the changes as Preview or Major version.

In both cases, major version is bumped (with the exception in preview where -preview string is appended to the version number) and build/release pipelines are notified about the availability of the new version (a small flag icon appears just below the task title in the pipeline). Mostly, you will roll out the changes as a Preview, try it out in few of the production pipelines and then re-publish it as a major version (Publish draft button in task group edit window will now be replaced with Publish preview button).

Publishing your changes replaces base task group with the new one. You are no longer permitted to update/modify the old task group. Further changes can only be made to the most recent version.

Useful Tips

• To find out where all the task group is referenced, select your task group from Task Groups page and go to References tab.
• To expose a variable inside the task group as configurable parameter, define it with $(ConfigVariableName) syntax.
• The scope of task groups is only till Project Level. To use them across projects, you have to export it as json and then import it in the new project. Import/Export of task group can be performed from Task Group page.

Summary

Task groups is a handy feature to encapsulate common build/release tasks into a single entity that can be reused across many pipelines (and other task groups as well). While smaller changes can easily be made and automatically applied to all the definitions that refer it, breaking changes can be rolled out in a lockstep manner with little impact on the running pipeline operations.

Thanks for reading this article! I hope you benefited from it.