Open in app

Sign in

Write

Sign in

Gitops your Terraform resources on Azure with Flux

Victor Muchiaroni
6 min readJan 4, 2024

--

Gitops is a really interesting approach to speed up the deliver of applications running under Kubernetes making it easier to manage the resources by centralizing the components in a git repository. To use this as a single source of truth for your infrastructure resources on Azure can also help to keep things organized and consistent.

There are lots of tools that can help us to achieve our goal to implement a consistent and centralized environment using the “Gitops way”. For this article, I have chosen Flux, which is a CNCF graduated project and has a useful terraform controller that can manage our resources over different cloud providers. In our scenario, Flux will check for changes in a git repository where our terraform definitions are and reconcile them by adding, changing or deleting Azure cloud resources accordingly.

There are some prerequisites that are needed to be fulfilled in order for our premise to work:

  • Some previous Flux or Gitops knowledge may help on this reading. I recommend the Linuxfoundation free Gitops training https://training.linuxfoundation.org/training/introduction-to-gitops-lfs169/;
  • A git repository to host our flux and terraform configuration. For this article, all flux and terraform configuration will be available on https://github.com/vimuchiaroni/flux-terraform-azure. In this case, we are using Github, but it also works with different providers like Gitlab or Bitbucket.
  • An Azure account;
  • An Azure Service principal with Contributor Role;
  • A working Kubernetes cluster;
  • Flux installed;
  • Terraform controller for Flux installed;

Working on the prerequisites

Flux requires a Kubernetes cluster where its controllers will run. To make our life a little bit easier, I have automated the deployment and configuration of our cluster initial setup by using terraform itself. For this, we will create a local Kubernetes with Kind, bootstrap the flux installation using our previously created Github repository and install the terraform controller. All the code for this, can be found here: https://github.com/vimuchiaroni/kind-terraform. If you prefer to install the Flux components manually in an existing cluster, you can follow the official documentation to bootstrap Flux here and to install the terraform controller here.

Since AzureRM provider for terraform requires authentication(https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs#authenticating-to-azure), we will need, for this example, to create a service principal with Contributor role to be able to deploy our cloud resources. In order to do that, we can run the following az cli commands:

# export SUBSCRIPTION_ID=11111-2222-3333-4444
# az account set --subscription=$SUBSCRIPTION_ID
# az ad sp create-for-rbac --role="Contributor" --scopes="/subscriptions/$SUBSCRIPTION_ID" --name sp-flux-terraform-poc

The last az cli command will display an output similar to:

{
  "appId": "00000000-0000-0000-0000-000000000000",
  "displayName": "sp-flux-terraform-poc",
  "name": "0000-0000-0000-0000-000000000000",
  "password": "0000-0000-0000-0000-000000000000",
  "tenant": "00000000-0000-0000-0000-000000000000"
}

These values map to the Terraform variables like so:

  • appId is the client_id defined above.
  • password is the client_secret defined above.
  • tenant is the tenant_id defined above.

We will need to save the client id and client secret together with our Tenant and subscription id to use them in the next steps.

For flux to be able to manage our terraform cloud resources on Azure from our cluster, we will need to create a Kubernetes Secret with the service principal information we got earlier:

# export NAMESPACE=flux-system
# kubectl create namespace $NAMESPACE
# kubectl create secret generic azure-auth \
 --from-literal ARM_SUBSCRIPTION_ID=$SUBSCRIPTION_ID \
 --from-literal ARM_CLIENT_ID=<client_id> \
 --from-literal ARM_CLIENT_SECRET=<client_secret> \
 --from-literal ARM_TENANT_ID=<tenant_id> \
 --namespace $NAMESPACE

Note: Creating the secret manually is the fastest way to demonstrate the terraform controller. For production use, I would recommend to Gitops your kubernetes secrets(https://fluxcd.io/flux/security/secrets-management/). Secrets in Gitops is a different topic and I will, probably, cover that with more details in another article.

Gitops your infrastructure with terraform

Once the prerequisites have been completed, we can declare our terraform resources and configure them in a way that the Flux controller will be able to plan and apply those. When you bootstrap Flux and install the terraform controller in your cluster, it deploys some CustomResourceDefinitions that will allow us to create specific Kubernetes objects. Below are the examples we are going to use:

apiVersion: infra.contrib.fluxcd.io/v1alpha2
kind: Terraform
metadata:
  name: my-terraform-resource
  namespace: flux-system
spec:
  interval: 1m
  approvePlan: auto
  destroyResourcesOnDeletion: true
  path: ./terraform/azure/storage-account
  sourceRef:
    kind: GitRepository
    name: flux-system
  runnerPodTemplate:
    spec:
      image: docker.io/vimuchiaroni/tf-az-cli:1.0
      env:
        - name: ARM_SUBSCRIPTION_ID
          valueFrom:
            secretKeyRef:
              name: azure-auth
              key: ARM_SUBSCRIPTION_ID
        - name: ARM_CLIENT_ID
          valueFrom:
            secretKeyRef:
              name: azure-auth
              key: AZURE_CLIENT_ID
        - name: ARM_CLIENT_SECRET
          valueFrom:
            secretKeyRef:
              name: azure-auth
              key: AZURE_CLIENT_SECRET
        - name: ARM_TENANT_ID
          valueFrom:
            secretKeyRef:
              name: azure-auth
              key: AZURE_TENANT_ID

The above configuration will create a Terraform object that will be responsible for managing the terraform resources on cloud. It will be managed by flux when we place it under ./clusters/tfkind/terraform.yaml and will get our terraform resources under ./terraform/azure/storage-account in our git repository.

The repository structure will look like this:

├── README.md
├── clusters
│   └── tfkind
│       ├── flux-system
│       │   ├── gotk-components.yaml
│       │   ├── gotk-sync.yaml
│       │   └── kustomization.yaml
│       └── terraform.yaml
└── terraform
    └── azure
        └── storage-account
            └── main.tf

Once you push your objects to the git repository, you will be able to see flux and the terraform controller in action:

# kubectl get terraforms.infra.contrib.fluxcd.io -n flux-system 
NAME                    READY     STATUS                       AGE
my-terraform-resource   Unknown   Reconciliation in progress   44s

# kubectl get po -n flux-system my-terraform-resource-tf-runner 
NAME                              READY   STATUS    RESTARTS   AGE
my-terraform-resource-tf-runner   1/1     Running   0          5s

# kubectl logs -f -n flux-system my-terraform-resource-tf-runner
2024/01/04 18:24:23 Starting the runner... version main sha 09c1a41
{"level":"info","ts":"2024-01-04T18:24:37.681Z","logger":"runner.terraform","msg":"preparing for Upload and Extraction","instance-id":""}
{"level":"info","ts":"2024-01-04T18:24:37.694Z","logger":"runner.terraform","msg":"write backend config","instance-id":"","path":"/tmp/flux-system-my-terraform-resource/terraform/azure/storage-account","config":"backend_override.tf"}
{"level":"info","ts":"2024-01-04T18:24:37.694Z","logger":"runner.terraform","msg":"write config to file","instance-id":"","filePath":"/tmp/flux-system-my-terraform-resource/terraform/azure/storage-account/backend_override.tf"}
{"level":"info","ts":"2024-01-04T18:24:37.695Z","logger":"runner.terraform","msg":"looking for path","instance-id":"","file":"terraform"}
{"level":"info","ts":"2024-01-04T18:24:37.696Z","logger":"runner.terraform","msg":"creating new terraform","instance-id":"55fa27dd-e53e-4fe4-b8c3-b82e5ed08438","workingDir":"/tmp/flux-system-my-terraform-resource/terraform/azure/storage-account","execPath":"/usr/local/bin/terraform"}
{"level":"info","ts":"2024-01-04T18:24:37.704Z","logger":"runner.terraform","msg":"setting envvars","instance-id":"55fa27dd-e53e-4fe4-b8c3-b82e5ed08438"}
{"level":"info","ts":"2024-01-04T18:24:37.704Z","logger":"runner.terraform","msg":"getting envvars from os environments","instance-id":"55fa27dd-e53e-4fe4-b8c3-b82e5ed08438"}
{"level":"info","ts":"2024-01-04T18:24:37.705Z","logger":"runner.terraform","msg":"setting up the input variables","instance-id":"55fa27dd-e53e-4fe4-b8c3-b82e5ed08438"}
{"level":"info","ts":"2024-01-04T18:24:37.705Z","logger":"runner.terraform","msg":"mapping the Spec.Values","instance-id":"55fa27dd-e53e-4fe4-b8c3-b82e5ed08438"}
{"level":"info","ts":"2024-01-04T18:24:37.705Z","logger":"runner.terraform","msg":"mapping the Spec.Vars","instance-id":"55fa27dd-e53e-4fe4-b8c3-b82e5ed08438"}
{"level":"info","ts":"2024-01-04T18:24:37.705Z","logger":"runner.terraform","msg":"mapping the Spec.VarsFrom","instance-id":"55fa27dd-e53e-4fe4-b8c3-b82e5ed08438"}
{"level":"info","ts":"2024-01-04T18:24:37.706Z","logger":"runner.terraform","msg":"generating the template founds"}
{"level":"info","ts":"2024-01-04T18:24:37.706Z","logger":"runner.terraform","msg":"main.tf.tpl not found, skipping"}
{"level":"info","ts":"2024-01-04T18:24:37.706Z","logger":"runner.terraform","msg":"initializing","instance-id":"55fa27dd-e53e-4fe4-b8c3-b82e5ed08438"}
{"level":"info","ts":"2024-01-04T18:24:37.706Z","logger":"runner.terraform","msg":"mapping the Spec.BackendConfigsFrom","instance-id":"55fa27dd-e53e-4fe4-b8c3-b82e5ed08438"}
{"level":"info","ts":"2024-01-04T18:24:46.892Z","logger":"runner.terraform","msg":"workspace select"}
{"level":"info","ts":"2024-01-04T18:24:46.931Z","logger":"runner.terraform","msg":"creating a plan","instance-id":"55fa27dd-e53e-4fe4-b8c3-b82e5ed08438"}
{"level":"info","ts":"2024-01-04T18:24:52.837Z","logger":"runner.terraform","msg":"save the plan","instance-id":"55fa27dd-e53e-4fe4-b8c3-b82e5ed08438"}
{"level":"info","ts":"2024-01-04T18:24:52.885Z","logger":"runner.terraform","msg":"loading plan from secret","instance-id":"55fa27dd-e53e-4fe4-b8c3-b82e5ed08438"}
{"level":"info","ts":"2024-01-04T18:24:52.903Z","logger":"runner.terraform","msg":"running apply","instance-id":"55fa27dd-e53e-4fe4-b8c3-b82e5ed08438"}
azurerm_resource_group.example: Creating...
azurerm_resource_group.example: Creation complete after 2s [id=/subscriptions/a278ca6b-6478-4182-8090-1fcdfd72079f/resourceGroups/rg-fluxterraform]
azurerm_storage_account.example: Creating...
azurerm_storage_account.example: Still creating... [10s elapsed]
azurerm_storage_account.example: Still creating... [20s elapsed]
azurerm_storage_account.example: Creation complete after 30s [id=/subscriptions/a278ca6b-6478-4182-8090-1fcdfd72079f/resourceGroups/rg-fluxterraform/providers/Microsoft.Storage/storageAccounts/stfluxterraform]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
{"level":"info","ts":"2024-01-04T18:25:30.088Z","logger":"runner.terraform","msg":"creating outputs","instance-id":"55fa27dd-e53e-4fe4-b8c3-b82e5ed08438"}
{"level":"info","ts":"2024-01-04T18:25:30.416Z","logger":"runner.terraform","msg":"cleanup TmpDir","instance-id":"55fa27dd-e53e-4fe4-b8c3-b82e5ed08438","tmpDir":"/tmp/flux-system-my-terraform-resource"}

# kubectl get terraforms.infra.contrib.fluxcd.io -n flux-system
NAME                    READY   STATUS                                                         AGE
my-terraform-resource   True    No drift: main@sha1:39d1f53d45d8e0a83f84b9a026e538a97ada8685   15m

Note: In the example Terraform object, there are some specific configuration for Azure that I would like to highlight:

  • spec.runnerPodTemplate.spec.image: Since az cli is required in the environment where terraform will run, we need to create a custom image with it to run our terraform jobs in the Kubernetes cluster instead of the default one. In this case, I’ve built a custom image under my personal dockerhub repo docker.io/vimuchiaroni/tf-az-cli:1.0 which is public but, if you prefer to build your own image, you can use my Dockerfile as example here.
  • spec.runnerPodTemplate.spec.env: In this block, we will set our environment variables that terraform will use to authenticate with Azure(https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/guides/service_principal_client_secret#configuring-the-service-principal-in-terraform). We will refer to our previous created Kubernetes secret containing the necessary credentials to our Azure account.

References:
https://www.redhat.com/en/topics/devops/what-is-gitops

https://fluxcd.io/blog/2022/09/how-to-gitops-your-terraform/

Gitops
Flux
Kubernetes
Terraform
Azure

--

--

Victor Muchiaroni

Written by Victor Muchiaroni

16 Followers

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams