A Terraform, AKS and Application Gateway Tutorial — Part 6

5 min readSep 29, 2023

Introduction

In part 5 we introduced the Secret Store CSI Driver to import secrets as file mounts and environment variables into our pods.

In part 6 (the final bit) we’ll combine the yaml files from parts 1, 2, 4 and 5 into a Helm chart for easier deployment. For more information on Helm see https://helm.sh.

Full Code

For those of you who just want to see the final code; it’s here, part6 :-)

If you’re interested in how these lessons can be combined together in a more production ready pipeline then take a look at the demo on my github which builds on these concepts further; demo 1.

Packaging Our Yaml Files With HELM

1. Assumptions

You’ve completed parts 1, 2, 3 (the key vault bit) 4 and 5, tested your deployment and have nginx accessible over HTTPS via application gateway.

You have Helm installed.

2. Create Helm Chart Template

Run the code below to create a template helm structure for us to update.

cd helm
helm create titan

We can delete the helm/titan/templates folder as we won’t need that.

# Ensure you are in the helm directory.
Remove-Item -Path "titan\templates" -Recurse -Force

Run the code below to build the structure out further. You’ll see we are going to create separate charts for prereqs and the application (titan).

# Ensure you are in the helm directory.

# Define the directory structure
$rootPath = "titan"  # Replace with your desired path
$structure = @(
  "charts/prereqs/templates/",
  "charts/prereqs/Chart.yaml",
  "charts/prereqs/values.yaml",
  "charts/titan/templates/",
  "charts/titan/Chart.yaml",
  "charts/titan/values.yaml"
)

# Create the directory structure and empty files
foreach ($item in $structure) {
  $fullPath = Join-Path -Path $rootPath -ChildPath $item
  # Check if the item ends with '/' (indicating a directory) and create a directory if needed
  if ($item.EndsWith('/')) {
    $null = New-Item -ItemType Directory -Path $fullPath -Force
  }
  else {
    # Create an empty file if the item is not a directory
    $null = New-Item -ItemType File -Path $fullPath -Force
  }
}

3. PreReqs Chart

Copy the k8s/6-cluster-issuer.yaml file into helm/titan/charts/prereqs/templates
Copy the k8s/8-csi-driver.yaml file into helm/titan/charts/prereqs/templates
Update helm/titan/charts/prereqs/Chart.yaml with the values below.

apiVersion: v2
name: prereqs
description: A helm chart for prereqs.
version: 0.0.1
appVersion: 0.0.1

4. Titan Chart

Copy the k8s/2-nginx-80-service.yaml file into helm/titan/charts/titan/templates
Copy the k8s/7-nginx-443-ingress-cert-manager.yaml file into helm/titan/charts/titan/templates
Copy the k8s/9-nginx-80-pod-secrets.yaml file into helm/titan/charts/titan/templates
Update helm/titan/charts/titan/Chart.yaml with the values below.

apiVersion: v2
name: titan
description: A helm chart for titan.
version: 0.0.1
appVersion: 0.0.1

5. Root Folder

Update helm/titan/Chart.yaml with the values below.

apiVersion: v2
name: titan
description: A helm chart for the titan app microservice.
version: 0.0.1
appVersion: 0.0.1

Delete the contents of helm/titan/values.yaml. We’ll build it up later when we start parameterising some of our values.

Testing Our Helm Chart

We’re now ready to test the changes we’ve made.

1. Dry Run

Run the following command to do a dry run and verify your chart files. This will show you the amalgamated config from all the manifests. This is what will apply.

# Ensure you are in the helm directory.
kubectl delete namespace titan
kubectl delete clusterissuer letsencrypt-issuer
helm install --debug --dry-run titan ./titan

2. Install the Chart

Run the following command to install the chart.

# Ensure you are in the helm directory.
helm install titan ./titan --namespace titan --create-namespace --wait

3. Test

At this point you can run the previous tests to exec onto the pod and check the secrets are there, and fire up a browser and confirm the app is online :-)

The following commands can be used to interrogate the helm installation.

helm list -A
helm status titan -n titan
helm get manifest titan -n titan

3. Uninstall

Run the following command to uninstall the helm chart.

helm uninstall titan -n titan

Parameterising Our Helm Chart

One of the key benefits of Helm is the power of parametrising values. We’ll parameterise a few values below to show you how this works. This becomes even more useful when you start deploying charts via pipelines and you need to switch values between different environments, maybe based on a branch name.

1. Updating the Files

Update helm/titan/values.yaml with the values below. You’ll need to update all these values to reflect your own settings.

# Global values; must use 'global' heading.
global:
  certManagerEmail: "rhod3rz@outlook.com"
  certManagerServer: "https://acme-staging-v02.api.letsencrypt.org/directory"
  # This is the aks_uai_agentpool_client_id value.
  userAssignedIdentityID: "66191928-e4d1-4af5-991f-d24e2563aea3"
  tenantId: "73578441-dc3d-4ecd-a298-fc5c6f40e191"
  keyvaultName: "kv-prd-titan-230101"

# Service specific values; must use chart/folder name & referenced as {{ .Values.hostname }}.
titan:
  hostname: "prd.rhod3rz.com"

Update helm/titan/charts/prereqs/templates/6-cluster-issuer.yaml with the values below. Notice how we are now using parameters for email and server values.

---
# CLUSTERISSUER
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-issuer
spec:
  acme:
    email: {{ .Values.global.certManagerEmail }}
    server: {{ .Values.global.certManagerServer }}
    privateKeySecretRef:
      name: letsencrypt-issuer
    solvers:
    - http01:
        ingress:
          class: azure/application-gateway
---

Update helm/titan/charts/prereqs/templates/8-csi-driver.yaml with the values below. Again notice the parameters we are now using.

---
# SECRET PROVIDER CLASS
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: spc-kv-prd-titan-230101
  namespace: titan
spec:
  provider: azure

#================================================================================================
# Mounted Files - These secrets can be mounted in pods as files.
#================================================================================================

  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "true"
    userAssignedIdentityID: {{ .Values.global.userAssignedIdentityID }}
    tenantId: {{ .Values.global.tenantId }}
    keyvaultName: {{ .Values.global.keyvaultName }}
    objects: |
      array:
        - |
          objectName: titan-username
          objectType: secret
          objectVersion: ""
        - |
          objectName: titan-password
          objectType: secret
          objectVersion: ""

#================================================================================================
# Environment Variables - These secrets can be mounted in pods as environment variables.
#================================================================================================

  secretObjects:
  - secretName: so-titan
    type: Opaque
    data:
    - objectName: titan-username
      key: titan_username
    - objectName: titan-password
      key: titan_password

---

And finally update helm/titan/charts/titan/templates/7-nginx-443-ingress-cert-manager.yaml with the values below. Again notice the parameters we are now using.

---
# INGRESS
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx
  namespace: titan
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    appgw.ingress.kubernetes.io/ssl-redirect: "true"
    cert-manager.io/cluster-issuer: letsencrypt-issuer
spec:
  tls:
  - hosts:
    - {{ .Values.hostname }}
    secretName: {{ .Values.hostname }}
  rules:
  - host: {{ .Values.hostname }}
    http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: nginx
            port:
              number: 80
---

2. Dry Run

Run the following command to do a dry run and verify your chart files.
If you review the output you’ll see that helm has substituted your parameterised values with the real values from values.yaml.

# Ensure you are in the helm directory.
helm install --debug --dry-run titan ./titan

3. Install the Chart

Run the following command to install the chart.

# Ensure you are in the helm directory.
helm install titan ./titan --namespace titan --create-namespace --wait

4. Test