Terraform Plan With Azure DevOps YAML Pipelines — Part 1

Photo by Jeremy Bishop on Unsplash

An introduction to YAML pipelines for Terraform in Azure DevOps

In the last several months, Microsoft has made quite a few improvements to their YAML pipelines. I’d been waiting on the YAML pipelines to mature before jumping in, and in early December a fellow engineer and I worked together on implementing YAML for one of his projects.

After designing build and release pipelines for Terraform through the ADO UI for several months I was pleasantly surprised by how far the YAML schema had come since the last time I’d looked into using them. YAML pipelines are still limited to Build Pipelines only, but Microsoft is planning to make them available for Release Pipelines in the near future.

I decided it was time to make the switch to YAML pipelines for the Terraform pipelines for a couple reasons.

• YAML pipelines follow the DevOps pattern of Configuration as Code.
• It’s easier to reason about and connect with the Terraform code. The pipeline is embedded with the Terraform code and viewable in the code editor, rather than having to refer to the ADO UI.

Intro to YAML Pipelines

The YAML schema reference is helpful when building pipelines. There’s still some missing pieces in the documentation, but I’ve found that Microsoft has been pretty responsive if you have a question about the syntax.

To keep the YAML file organization as DRY as possible, I take advantage of step templates. Most of the steps for the pipelines are stored in a Templates repository. In the individual Terraform pipelines, I just reference the templates I need for that stack.

Terraform Project Directory

Here is a sample azure-pipelines.yml file that would be embedded with the Terraform code. I’ll break down each section below.

Build Name

name: $(BuildDefinitionName).$(DayOfYear)$(Rev:.r)

This is the build number format that will appear in ADO when the builds are run.

Resources

resources:
  repositories:
    - repository: Templates
      type: git
      name: Templates

The resources block is how we tell ADO what repository to find the template files in.

Pool

pool:
  name: Terraform Agents

The pool block is how we tell ADO what agent pool to use. In this case I’m using a custom agent pool called Terraform Agents, but the ADO hosted ubuntu 16.04 could be used as your agent pool as well.

Trigger

trigger:
  batch: true
  branches:
    include:
      - master
      - feature/*
  paths:
    include:
      - 'SOMEPATH/DEV/AKS'

The trigger block is where we tell ADO what will trigger the pipeline. I set batch: true so that if multiple commits come in within a short period of time, the individual build pipelines will be processed one at a time rather than all at once. This is important to keep multiple builds from stepping on each other when they lock the terraform state.

I also have a trigger path. This is because repos may have more than one Terraform stack in it and we only want to trigger this pipeline for this path.

Variables

variables:
- name: state.key
  value: 'TFSTATE'
- name: terraform.path
  value: 'SOMEPATH/DEV/AKS'
- group: 'KEY-VAULT-VAR-GROUP'
- group: 'STANDARD-VAR-GROUP'

In the variables block I have two named variables specific to this pipeline. I also include two variable groups, one which is backed by an Azure KeyVault that contains secrets for the terraform plan and a standard ADO variable group.

The variable groups have to be defined within the ADO UI before you can use them in your pipelines.

Steps

steps:
- checkout: self
  clean: true
  persistCredentials: true
- template: ADO/Terraform-Build-Readiness.yml@Templates
- template: ADO/DEV-TF-plan.yml@Templates
- template: ADO/Publish-Terraform-Artifact.yml@Templates
- template: ADO/Publish-Plan-To-Wiki.yml@Templates

The steps block is where the tasks for this pipeline are defined.

In the checkout step we are telling ADO how to handle checking out the source code.

The actual work of the pipeline is done in the remaining steps. I have several template steps, which I’ll describe in future posts.

The most important template reference here is the DEV-TF-plan.yml template. I’ll break that and the Publish-Terraform-Artifact.yml template down in the next post.

Conclusion

This is just a quick introduction to the Azure Devops YAML pipelines and some of the schema attributes you can use in pipelines. In later posts I’ll go into more detail on what the templates are actually doing.

Going forward in 2019 I expect the YAML pipelines to continue improving at a rapid pace and we hopefully will see the availability of YAML for Release pipelines in the near future.

Links

Part 2 — Terraform Plan