Complex Infrastructure as Code via Azure Devops YAML Pipeline

Deploy ARM Templates via Azure Devops YML Pipeline to Multiple Azure Environments

Photo by Clark Tibbs on Unsplash

Introduction

These days we can define infrastructure as code (ARM template)and CI/CD pipeline as code (YAML pipelines) in the Azure ecosystem. But why do we want to do these tasks as code?

There are good reasons apart from just a cool thing to do.

• Consistency. The outcome in the production environment should be exactly the same as the dev environment.
• Scalability. Imagine managing 20+ services for a large solution. You wouldn’t want to do it on the UI.
• Code review. Because they are done via code now. It can fit into the standard code review process just like the application code.

It’s relatively straight forward to deploy a simple ARM template to a single Azure environment via a YAML pipeline. But in a commercial project, it’s most likely more complex than that.

Mission Statement

At the end of this article, you should be able to:

Deploy complex ARM templates with dependencies between resources, to multiple environments.

Note: if you would like to see the full files as a reference, they are available in my GitHub repo.

Hypothesis Application

The hypothesis application is simple: an Azure function that reads a secret named Secret (yes I’m that creative) from an Azure key vault and then displays it.

For such a simple project, we need to deploy the following resources:

• An Azure function app that runs business logic.
• A storage account that is required for function apps.
• Application Insights for monitoring.
• An Azure key vault to store sensitive data.

The tricky part is the dependency between these resources. The function app should be created last because we need to set its app settings to use app insights and the key vault.

However, the key vault needs the function to exist with an identity so that it can grant the function app access to the secrets…

Which one to create first, then? The function app, or the key vault? This becomes a chicken first or egg first problem.

Or does it?

The ARM Template

The solution is to create the function with a system-assigned identity before the key vault so that we can assign the correct permissions to the identity. Then we can go back to the function to set the secret URI in the app settings. All these can be done in the same ARM template.

Let’s go through the sections in the ARM template.

Pre-warning: the ARM template is quite long and can look confusing at first. Bear with me, it should all make sense in the end.

Parameters and Variables

It is called infrastructure as code. And in code, there are parameters and variables. In an ARM template, parameters let the caller passes in values while variables are used to store long expressions to make the template more readable.

In this project, the following parameters and variables are defined.

The Azure Function App

We have covered the code bit, now let’s move to the infrastructure.

An Azure Function App is a special kind of a web app with some dependencies.

Compulsory dependencies:

• An Azure storage account.
• An app service plan.

Optional dependencies:

• Application insights.

We can specify its dependencies via the dependsOn property to instruct Azure to create dependent resources first.

All these resources could look overwhelming if you’re not familiar with Azure Function Apps. You might want to check out part 1 of my series about Azure Durable Functions.

A Practical Guide to Azure Durable Functions — Part 1: Introduction

Now it’s time to create the key vault. The key vault depends on the function app so that the system assigned identity exists when we configure the access policy.

Last but not least, go back to the function and configure it to read secrets from the key vault. It might seem a little strange that app settings are resources on their own. But it’s really handy because we can configure it to depend on the function app and the key vault.

Now we have set up the infrastructure as code. What about the pipeline to deploy it?

The YAML Pipeline

Variable Groups

The goal is to be able to deploy to multiple environments. How do we let the pipeline know what value to use for different environments?

Azure Devops lets users define variable groups. We’ll define a group for each environment. Click Library from the menu on the left-hand side, create two groups for dev and test environment.

Later in the pipeline, we’ll pass the Environment variable to the ARM template to create resources with the environment on their names.

Environment

This refers to Azure DevOps environments. Click on Environments from the menu on the left-hand side. Create dev and test environments.

This feature enables fine control of different environments. For example, developers may have Administrator access to the dev environment, but only Reader access to the production environment.

It also provides an approval gateway to the pipelines. A deployment to the test environment may require approval from the QA group. In this example, I placed an approval requirement in the test environment. Any deployment to the test environment should be approved by me.

The Pipeline

The YAML pipeline is organized into stages.

The very first stage is a publishing artifact stage to make the ARM template available to the pipeline. Note the trigger on the very top? It means that this pipeline will be automatically triggered by changes in the master branch.

The second stage deploys the ARM template to the dev resource group.

A few points to note here:

• The variable group DevGroup is specified at the stage level. The variables in that group become available in this stage. Reference the variables via the $(variableName) format.
• The environment is set to DevEnv created earlier at the job level. I’m not sure why I can’t set it at the stage level. If you happen to know, please leave a comment. The idea is that the test stage will be protected by the approval gateway defined earlier.
• Deployment mode is set to Incremental. This means that the pipeline will not try to delete resources that are not listed in the ARM template. It’s the safer option as devs may manually create resources to run experiments. But it does mean that they need to remember to clean up the extra resources when they are no longer needed.

The third stage looks very similar to the second, the only difference is the variable group, the environment, and the target resource group.

The Result

Now that everything is ready. Let’s deploy to Azure.

The first thing to notice is that the dev environment deployment just runs. But the test environment deployment is pending on approval. The approval gateway on the Environments is working.

Click Approve and the resources will be deployed to the test environment, too.

The deployed resource should have the correct environment name on them.

Head to the key vault, make sure the function’s identity has Get and List permission to the secrets. If you want to test the function later, you may want to add a Secret for the function to consume while you are here.

Lastly, check the function app’s settings. Make sure the key vault reference and app insights instrumentation key are correctly configured.

Summary

That’s it. We have successfully deployed Azure resources with dependencies via the Azure DevOps pipeline, both as code.

Feel free to deploy some code to the function app. It should be able to read the Secret value from the key vault.

The “as code” version certainly has its benefits and is the way going forward — Azure DevOps encourages to use the YAML pipeline as opposed to the classic pipeline with user-friendly UI.

But it has a steep learning curve at the beginning. Here are some tips to help you get started:

• Do not try to create ARM templates from scratch. Create resources in the dev environment’s resource group. Then export the ARM template from the resource group. The exported template may not be exactly how you want it. But it’s a good start.
• It’s extremely easy to mess up all those square brackets, brackets, single quotes, and double quotes in the ARM templates. Visual Studio’s IntelliSense is a pretty good tool to help with that. However, sometimes it gets the parameters of the APIs wrong. So don’t blindly trust it.
• Miss the easy-to-use dropdown options on the classic Azure DevOps pipelines? Edit the YAML pipeline on Azure DevOps and check out the assistant on the top right corner. It’s not as convenient as the UI, but it’s not bad.
• Not sure what parameters a task has? Make sure your cursor is at the correct place and press CTRL + space (just like you do in Visual Studio), available parameters will be listed.

Finally, something for you to think about: the duplication in the dev and test stages in the YAML pipeline may look fine with just two environments. What if there are also UAT, staging, and production environments? It’s obviously not ideal to duplicate those codes for every environment. So, what can you do to make it more readable and easy to maintain?

Hint: google “Azure Devops YAML template”.