How to Implement an Azure Landing Zone Using the Microsoft Cloud Adoption Framework (CAF) — Part 3

Hai Nguyen
The Factory
Published in
14 min readJul 28, 2021

How to deploy your landing zone using Azure DevOps Pipeline

Part 1 — Azure Landing Zone Architecture
Part 2 — Azure Landing Zone Level 0 — Launchpad
Part 3 — Azure Landing Zone DevOps agent and CI/CD pipeline
Part 4 — Azure Landing Zone Level 1
Part 5 — Azure Landing Zone Level 2

Introduction

In our previous blogs, we introduced you to the Landing Zone, how it works, what it looks like, and how to deploy it from your local machine. In this blog, I will show you how we can use Azure DevOps to automatically deploy your Landing Zone using a CI/CD pipeline.

1. Introduction to Azure DevOps

Azure DevOps is a centralized tool for your project. It includes Sprint planning, Kanban board, Repositories, Pipelines, Test Plans, and Artifacts. Basically everything you needed to work on development projects. For the purpose of this blog, we will only utilize the Repositories and Pipelines functionality of Azure DevOps.

2. Introduction to Azure Pipelines

Azure Pipelines automatically builds and tests code projects to make them available to others. Azure Pipelines combines continuous integration(CI) and continuous delivery(CD) to constantly and consistently test and build your code and ship it to any target.

For this project, we use Azure Pipelines to do the deployment for us. It will do 2 things in the following order. First, with the help of Rover, on each pull request to the main branch of the caf-configuration repo, a terraform plan will be created. This will validate the Terraform code and provide an overview of which resources will be created, modified, or removed when you execute the terraform apply. Once the pull request is approved and merged to the main branch, the Rover container will execute a Terraform apply command. This will apply the plan from the previous step and make the desired changes to your resources.

To be able to make use of the Azure Pipelines, we first need to host our own Azure DevOps agents.

3. Azure DevOps agents

There are few ways you can authenticate Azure DevOps agents for your deployment: using an Azure AD Application, using a Managed Identity with a single VM, or using a Managed Identity with a Virtual Machine Scale Set.

3.1 Azure AD Application

With Azure AD Application, you create a registered application in Azure AD, then you paste over the secret/credential of the application to the agent to assume the identity of the application. Followingly you can give the Service Principal of the AD Application the necessary permissions to create resources on your behalf. The downside is that you will need to have permission to register the Applications in Azure AD, and setting up the process is more complicated. Therefore I do not recommend this route unless you have a specific reason to do so.

3.2 Managed Identity with a single VM

Using Managed Identity, you can simply create a user-assign Managed Identity, and attach it to the VM that you spin up for the Azure DevOps agent. That’s it, you can then give the managed identity permission to create resources in any subscription within your tenant, and even Management Group if you want to, and the agent will have permission to do so.

3.3 Virtual Machine Scale Set

The two methods above have downsides. Noticeably is the High Availability of the agents. Since the agent runs on a single VM, if they become unhealthy, down goes your pipeline. If you use the pipeline itself to also deploy the agents, it will become a very tiresome thing to fix. In addition, the Virtual Machine will be idle for most of the time, wasting precious resources.

Due to those negative points, the best way to create the agents for your pipeline is to create the Virtual Machine Scale Set (VMSS) instead. It still uses user-assigned Managed Identity, but it can scale up and down based on workload, and you can simply terminate the VM and let it create a new one if one of the VM’s becomes unhealthy. This is also the way to create a minimum amount of overhead, so this is the method we will use to create the Azure DevOp agents

4. Implement Azure DevOps Agents module for your Landing Zone

caf-terraform-landingzones provides blueprints for the first 2 scenarios, but not for the case of VMSS. So to integrate VMSS as the agent deployment method for our pipeline, we need to create our own Terraform module for it.

Head to theterraform-landingzones repository, head to the add-ons folder and delete all the folders there, then you can create a new folder, let's call it azure_devops_agent, we will build our VMSS Terraform module here.

First, create the files in the azure_devops_agent folder as shown below:

We will go through each file and populate them, let’s start first with variables.tf file.

variables.tf

Here is where you define the variables needed to deploy the azure_devops_agent module, you will need:

  • tfstate file of launchpad that was deployed in the previous blog
  • landingzone configuration
  • resource group to create and deploy the VMSS devops agents on
  • some extra custom tags if need be
  • vmss configuration
  • VNET data to deploy the vmss into, this information will be extract from the remote statefile of launchpad
  • managed identity data to assign to the vmss, this information will be extracted from the remote statefile of launchpad

With that data, you can populate the variables.tf file as shown below:

variable tfstate_storage_account_name {}
variable tfstate_container_name {}
variable tfstate_resource_group_name {}
variable global_settings {
default = {}
}
variable landingzone {
}
variable tags {
default = null
}
variable resource_groups {
default = {}
}
variable vmss {
default = {}
}

backend.azurerm

Here you add the backend for the Terraform remote state file since we keep all the tfstate files in our Azure storage account. We add the backend with this:

terraform {
backend "azurerm" {
}
}

now we move to main.tf file:

main.tf

There is not much to go through here. This is where you define all the Terraform providers that are needed for the Terraform module deployment, so just fill in the file as shown below:

terraform {
required_providers {
azurerm = {
source = "hashicorp/azurerm"
version = "2.40.0"
}
azuread = {
source = "hashicorp/azuread"
version = "~> 1.0.0"
}
random = {
source = "hashicorp/random"
version = "~> 2.2.1"
}
external = {
source = "hashicorp/external"
version = "~> 1.2.0"
}
null = {
source = "hashicorp/null"
version = "~> 2.1.0"
}
tls = {
source = "hashicorp/tls"
version = "~> 2.2.0"
}
azurecaf = {
source = "aztfmod/azurecaf"
version = "~> 1.1.0"
}
}
required_version = ">= 0.13"
}
provider "azurerm" {
features {
key_vault {
purge_soft_delete_on_destroy = true
}
}
}
data "azurerm_client_config" "current" {}locals {
tfstates = merge(
map(var.landingzone.key,
map(
"storage_account_name", var.tfstate_storage_account_name,
"container_name", var.tfstate_container_name,
"resource_group_name", var.tfstate_resource_group_name,
"level", var.landingzone.level,
"subscription_id", data.azurerm_client_config.current.subscription_id
)
)
,
data.terraform_remote_state.remote[var.landingzone.global_settings_key].outputs.tfstates
)
}

locals.current_tfstates.tf

Here is where you extract the data from the tfstate file of the lower level of the Landing Zone, in this case, it is the tfstate file of the Launchpad that you deployed earlier. This data will be merged with other custom variables that you can paste on the module to create ready-to-use local Terraform data for module consumption.

let’s fill them in now:

locals {
landingzone = {
current = {
storage_account_name = var.tfstate_storage_account_name
container_name = var.tfstate_container_name
resource_group_name = var.tfstate_resource_group_name
}
}
}
data "terraform_remote_state" "remote" {
for_each = try(var.landingzone.tfstates, {})
backend = var.landingzone.backend_type
config = {
storage_account_name = local.landingzone[try(each.value.level, "current")].storage_account_name
container_name = local.landingzone[try(each.value.level, "current")].container_name
resource_group_name = local.landingzone[try(each.value.level, "current")].resource_group_name
key = each.value.tfstate
}
}
locals {
landingzone_tag = {
"landingzone" = var.landingzone.key
}
tags = merge(var.tags, local.landingzone_tag, local.global_settings.tags, { "level" = var.landingzone.level }, { "environment" = local.global_settings.environment })global_settings = data.terraform_remote_state.remote[var.landingzone.global_settings_key].outputs.global_settingsremote = {
managed_identities = {
for key, value in try(var.landingzone.tfstates, {}) : key => merge(try(data.terraform_remote_state.remote[key].outputs.managed_identities[key], {}))
}
vnets = {
for key, value in try(var.landingzone.tfstates, {}) : key => merge(try(data.terraform_remote_state.remote[key].outputs.vnets[key], {}))
}
}
}

vmss.tf

This is where you will deploy the resources of VMSS itself, to make it in-inline with other resources of the Landing Zone module, you will need to utilize azurecaf_name module as well to generate all the resource names you need here. So all the Terraform resources you will need to create include:

  • azurecaf_name for "vmss", "nic", and "rg"
  • "rg" - resource group for "vmss" module
  • "nic" - network interface card that can be used by "vmss"
  • "vmss" - the terraform vms module, in our case we will use "azurerm_linux_virtual_machine_scale_set" from azurerm provider

with that in mind, let’s create the module with the below resources:

resource "tls_private_key" "ssh" {
algorithm = "RSA"
rsa_bits = 4096
}
resource "azurecaf_name" "vmss" {
name = var.vmss[var.landingzone.level].virtual_machine_settings[var.vmss[var.landingzone.level].os_type].name
resource_type = "azurerm_linux_virtual_machine_scale_set"
prefixes = [local.global_settings.prefix]
random_length = local.global_settings.random_length
clean_input = true
passthrough = local.global_settings.passthrough
use_slug = local.global_settings.use_slug
}
resource "azurecaf_name" "nic" {
for_each = try(var.vmss[var.landingzone.level].networking_interfaces, {})
name = each.value.nameresource_type = "azurerm_network_interface"
prefixes = [local.global_settings.prefix]
random_length = local.global_settings.random_length
clean_input = true
passthrough = local.global_settings.passthrough
use_slug = local.global_settings.use_slug
}
resource "azurecaf_name" "rg" {
name = var.resource_groups[var.vmss[var.landingzone.level].resource_group_key].name
resource_type = "azurerm_resource_group"
prefixes = [local.global_settings.prefix]
random_length = local.global_settings.random_length
clean_input = true
passthrough = local.global_settings.passthrough
use_slug = local.global_settings.use_slug
}
resource "azurerm_resource_group" "rg" {
name = azurecaf_name.rg.result
location = local.global_settings.regions[local.global_settings.default_region]
tags = local.tags
}
resource "azurerm_linux_virtual_machine_scale_set" "vmss" {
name = azurecaf_name.vmss.result
resource_group_name = azurerm_resource_group.rg.name
sku = var.vmss[var.landingzone.level].virtual_machine_settings[var.vmss[var.landingzone.level].os_type].size
admin_username = var.vmss[var.landingzone.level].virtual_machine_settings[var.vmss[var.landingzone.level].os_type].admin_username
admin_ssh_key {
username = var.vmss[var.landingzone.level].virtual_machine_settings[var.vmss[var.landingzone.level].os_type].admin_username
public_key = tls_private_key.ssh.public_key_openssh
}
custom_data = filebase64("scripts/install-docker.cloud-config")
instances = 0
lifecycle {
ignore_changes = [ instances, custom_data ]
}
# Recommended values from https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/scale-set-agents?view=azure-devops#create-the-scale-set
overprovision = false
location = local.global_settings.regions[local.global_settings.default_region]
tags = local.tags
source_image_reference {
publisher = var.vmss[var.landingzone.level].virtual_machine_settings[var.vmss[var.landingzone.level].os_type].source_image_reference.publisher
offer = var.vmss[var.landingzone.level].virtual_machine_settings[var.vmss[var.landingzone.level].os_type].source_image_reference.offer
sku = var.vmss[var.landingzone.level].virtual_machine_settings[var.vmss[var.landingzone.level].os_type].source_image_reference.sku
version = var.vmss[var.landingzone.level].virtual_machine_settings[var.vmss[var.landingzone.level].os_type].source_image_reference.version
}
dynamic "network_interface" {
for_each = var.vmss[var.landingzone.level].networking_interfaces
content {
name = azurecaf_name.nic[network_interface.key].result
primary = try(network_interface.value.primary, false)
ip_configuration {
name = network_interface.value.name
primary = try(network_interface.value.primary, false)
subnet_id = local.remote.vnets[network_interface.value.lz_key][network_interface.value.vnet_key].subnets[network_interface.value.subnet_key].id
}
}
}
os_disk {
storage_account_type = var.vmss[var.landingzone.level].virtual_machine_settings[var.vmss[var.landingzone.level].os_type].os_disk.storage_account_type
caching = var.vmss[var.landingzone.level].virtual_machine_settings[var.vmss[var.landingzone.level].os_type].os_disk.caching
}
identity {
type = var.vmss[var.landingzone.level].virtual_machine_settings[var.vmss[var.landingzone.level].os_type].identity.type
identity_ids = [
for key in var.vmss[var.landingzone.level].virtual_machine_settings[var.vmss[var.landingzone.level].os_type].identity.remote[var.landingzone.global_settings_key].managed_identity_keys :
local.remote.managed_identities[var.landingzone.global_settings_key][key].id
]
}
}

Notice custom_data = filebase64("scripts/install-docker.cloud-config")? It is used to add Docker to the agent when bootstrapping it, so head to install-docker.cloud-config, and fill in the cloud-config for it:

#cloud-config# This is needed because of https://github.com/Azure/WALinuxAgent/issues/1938#issuecomment-657293920
bootcmd:
- mkdir -p /etc/systemd/system/walinuxagent.service.d
- echo "[Unit]\nAfter=cloud-final.service" > /etc/systemd/system/walinuxagent.service.d/override.conf
- sed "s/After=multi-user.target//g" /lib/systemd/system/cloud-final.service > /etc/systemd/system/cloud-final.service
- systemctl daemon-reload
apt:
sources:
docker.list:
source: "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"
keyid: 8D81803C0EBFCD88
packages:
- docker-ce
- docker-ce-cli
apt_upgrade: true
groups:
- docker

output.tf

This is optional since there won’t be a Terraform module to consume the output of the VMSS. For now, we can use output to print out some valuable information that gets generated after the Terraform apply

output vnets {
value = local.remote.vnets
sensitive = false
}
output global_settings {
value = local.global_settings
sensitive = true
}
output tfstates {
value = local.tfstates
sensitive = true
}

That’s it for the VMSS module, now let’s add the configuration to it and deploy the first VMSS for level 0.

5. Add level 0 VMSS Azure DevOps Agent and deploy it for your Landing Zone

First, head to terraform-landingzones repository, from landingzone/caf_launchpad folder add a new folder structure: devops_agentss/level0

In level0 folder add the files as shown below:

Yes, only 2 files this time. Let’s go through them and fill them in!

configuration.tfvars

Here, you pass in the variable related to the launchpad tfstate such as tfstate_storage_account_name, tfstate_container_name, and tfstate_resource_group_name. Then fill in some Landing Zone config related to the agent, and the resource_group name that will contain the VMSS. The configuration will look like below:

tfstate_storage_account_name = "oujestlevel0"
tfstate_container_name = "tfstate"
tfstate_resource_group_name = "ouje-rg-launchpad-level0"
landingzone = {
backend_type = "azurerm"
global_settings_key = "launchpad"
level = "level0"
key = "azdo-agent-level0"
tfstates = {
launchpad = {
level = "current"
tfstate = "caf_launchpad-caf_launchpad.tfstate"
}
}
}
resource_groups = {
rg1 = {
name = "devops-agents-level0"
}
}

Alright, let’s move on to the next file.

vmss.tfvars

This is where you can add the parameters needed to create the VMSS, such as os_type, size, os_disk,identity. Most of them are reusable for all agents, but you need to be mindful to choose the correct subnet, and more importantly, the correct UserAssiged managed identity, since that identity is what the agent will use to create resources for you, using RBAC.

so, fill in the configuration:

# Virtual machine scale set
vmss = {
# Configuration to deploy a bastion host linux virtual machine
level0 = {
resource_group_key = "rg1"
os_type = "linux"
# Define the number of networking cards to attach the virtual machine
networking_interfaces = {
nic0 = {
# Value of the keys from networking.tfvars
lz_key = "launchpad"
vnet_key = "devops_region1"
subnet_key = "release_agent_level0"
name = "0-release-agent-level0"
internal_dns_name_label = "release-agent-level0"
primary = true
}
}
virtual_machine_settings = {
linux = {
name = "release-agent-level0"
size = "Standard_D2s_v3"
admin_username = "adminuser"
disable_password_authentication = true
os_disk = {
caching = "ReadOnly"
storage_account_type = "StandardSSD_LRS"
}
source_image_reference = {
publisher = "Canonical"
offer = "UbuntuServer"
sku = "18.04-LTS"
version = "latest"
}
identity = {
type = "UserAssigned"
remote = {
launchpad = {
managed_identity_keys = [
"level0"
]
}
}
}
}
}
}
}

Now you are ready to deploy the agent for level 0. The first time you still need to do it locally, but after this, you can hand over deployment to the agent and let it deploy the resource for you via the pipeline that we will set up later.

Run the following commands:

rover loginaz account set — subscription "<subscription-id>"rover -lz /tf/caf/terraform-landingzones/landingzones/caf_launchpad/add-ons/azure_devops_agent \
-var-folder /tf/caf/caf-configuration/landingzone/caf_launchpad/devops_agents/level0/ \
-a plan \
-l level0 \
-env production \
-tfstate "azure_devops_agent-level0.tfstate"

Check the Terraform plan to see if there is an error. If there are none you can proceed with this command:

rover -lz /tf/caf/terraform-landingzones/landingzones/caf_launchpad/add-ons/azure_devops_agent \
-var-folder /tf/caf/caf-configuration/landingzone/caf_launchpad/devops_agents/level0/ \
-a apply \
-l level0 \
-env production \
-tfstate "azure_devops_agent-level0.tfstate"

The Rover should deploy the VMSS for you. Once the Terraform apply is done you can start the process of adding it to the Azure DevOps agent pool.

6. Adding VMSS to Azure DevOps agent pool

From the Azure DevOps portal, click on your Azure Landing Zone project. From the project portal click on Project Settings. Then from Pipelines → Agent pools, click on Add pool. An agent pool of type Azure virtual machine scale set has to be configured, with the right name: Self-hosted level <level> scale set

7. Adding pipelines

To set up the pipeline for your project, you can use the YAML schema to build your pipeline structure. Your YAML template will need to use caf-configuration and terraform-landingzones repositories, the Rover Image, variables groups for Rover version, and level0 agents information. The command to run Rover will be the same most of the time, so here we optionally create a job.yml file to add the script to use Rover to deploy the below script. azure-pipelines.yml then can refer to it for its job execution.

For each level of the Landing Zone, you create a pair of stages, 1 for plan and 1 for apply. Each state can contain multiple jobs to do a specific plan/apply of a subscription.

Alright, time for implementation, head to caf-configuration, and add the azure-pipelines.yml and job.yml as shown below:

Populate job.yml with this code block:

parameters:
- name: lz_dir
type: string
# example: terraform-landingzones/landingzones/caf_launchpad
- name: tfvars_dir
type: string
# example: caf-configuration/landingzone/launchpad/_launchpad_light.auto.tfvars
- name: level
type: number
values:
- 0
- 1
- 2
- 3
- 4
- name: job
type: string
default: ''
- name: displayName
type: string
default: ''
- name: launchpad
type: boolean
default: false
- name: action
type: string
values:
- plan
- apply
default: plan
- name: environment
type: string
values:
- production
- sandpit
default: sandpit
- name: pool_level
type: string
default: ''
- name: dependsOn
type: string
default: ''
- name: subscriptionId
type: string
default: ''
jobs:
- job: ${{ coalesce(parameters.job, format('level{0}_{1}', parameters.level, parameters.action)) }}
variables:
${{ if eq(parameters.launchpad, true) }}:
launchpad_opt: "-launchpad"
level_opt: ''
${{ if not(eq(parameters.launchpad, true)) }}:
launchpad_opt: ''
level_opt: "-level level${{ parameters.level }}"
${{ if eq(parameters.subscriptionId, '') }}:
subscription_cmd: "az account set --subscription '$(ARM_SUBSCRIPTION_ID)'"
tfstate_subscription_opt: ''
${{ if not(eq(parameters.subscriptionId, '')) }}:
subscription_cmd: "az account set --subscription '${{ parameters.subscriptionId }}'"
tfstate_subscription_opt: "-tfstate_subscription_id '$(ARM_SUBSCRIPTION_ID)'"
${{ if not(eq(parameters.dependsOn, '')) }}:
dependsOn: ${{ parameters.dependsOn }}
pool: $(AGENT_POOL_LEVEL${{ coalesce(parameters.pool_level, parameters.level) }})
displayName: ${{ coalesce(parameters.displayName, parameters.job, format('level{0} {1}', parameters.level, parameters.action)) }}
container: rover
steps:
- checkout: self
- checkout: terraform-landingzones
- bash: |
set -x
sudo chmod -R 777 /home/vscode

az login --identity -u '$(LEVEL${{ coalesce(parameters.pool_level, parameters.level) }}_AGENT_CLIENT_ID)'
${{ variables.subscription_cmd }}
/tf/rover/rover.sh \
-lz ${BUILD_REPOSITORY_LOCALPATH}/${{ parameters.lz_dir }} \
-a ${{ parameters.action }} \
-env ${{ parameters.environment }} ${{ variables.launchpad_opt }} ${{ variables.level_opt }} \
-var-folder ${BUILD_REPOSITORY_LOCALPATH}/${{ parameters.tfvars_dir }} \
-var="logged_user_objectId=$(LEVEL${{ parameters.level }}_AGENT_OBJECT_ID)" \
${{ variables.tfstate_subscription_opt }} -tfstate "$(basename ${{ parameters.lz_dir }})-$(basename ${{ parameters.tfvars_dir }}).tfstate"

Now fill in the azure-pipelines.yml with the code shown below:

resources:
repositories:
- repository: terraform-landingzones
type: git
name: terraform-landingzones
containers:
- container: rover
image: $(ROVER_IMAGE)
options: --user 0:0
variables:
- group: Rover-general
- group: Level0-agents
stages:
- stage: level0_plan
condition: eq(variables['Build.Reason'], 'PullRequest')
displayName: Plan Level 0
jobs:
- template: azure-pipelines-templates/job.yml
parameters:
lz_dir: terraform-landingzones/landingzones/caf_launchpad
tfvars_dir: caf-configuration/landingzone/caf_launchpad
level: 0
job: level0_plan_launchpad
displayName: Launchpad
launchpad: true
environment: production
- template: azure-pipelines-templates/job.yml
parameters:
lz_dir: terraform-landingzones/landingzones/caf_launchpad/add-ons/azure_devops_agent/
tfvars_dir: caf-configuration/landingzone/caf_launchpad/devops_agents/level0/
level: 0
job: level0_plan_devops_agents
dependsOn: level0_plan_launchpad
displayName: Level 0 agents
environment: production
- stage: level0_apply
condition: eq(variables['Build.SourceBranch'], 'refs/heads/main')
displayName: Apply Level 0
jobs:
- template: azure-pipelines-templates/job.yml
parameters:
lz_dir: terraform-landingzones/landingzones/caf_launchpad
tfvars_dir: caf-configuration/landingzone/caf_launchpad
level: 0
job: level0_apply_launchpad
displayName: Launchpad
launchpad: true
action: apply
environment: production
- template: azure-pipelines-templates/job.yml
parameters:
lz_dir: terraform-landingzones/landingzones/caf_launchpad/add-ons/azure_devops_agent/
tfvars_dir: caf-configuration/landingzone/caf_launchpad/devops_agents/level0/
level: 0
job: level0_apply_devops_agents
dependsOn: level0_apply_launchpad
displayName: Level 0 agents
action: apply
environment: production

This pipeline currently includes level0 Launchpad and the VMSS of Azure DevOps agent level0 plan and apply stage. You will need to expand upon this once you add more levels to your Landing Zone, such as level 1 and level2.

As you can see in the above code block, this pipeline depends on 2 variable groups, Rover-general and Level0-agents, so let's head to the Azure DevOps portal to fill them in before we deploy the pipeline.

In Azure DevOps portal, from your project head to Pipelines → Library → + Variable group. Add the Rover-general and Level0-agents as shown below:

For your implementation, fill in the value accordingly, such as the main subscription id you deploy your resource in, your tenant id, the Level0 agent client_id, and object_id, which you can find in the user-assign managed identity of the VMSS agent pool:

Now commit all the changes above. You are lready to deploy the pipeline.f

  1. From the Project portal, click on Pipelines → Pipelines → New pipeline and pick Azure Repos Git
  2. Select the caf-configuration repo.
  3. Select Existing Azure Pipelines YAML File
  4. You can now pick an azure pipeline’s YAML file to run as a pipeline.

Conclusion

Setting up a proper CI/CD pipeline is arguably one of the most important aspects of your Landing Zone project. Now that is done, you can utilize Azure DevOps to deploy the last 2 levels of our Landing Zone project.

--

--