Open in app

Sign in

Write

Sign in

A Terraform, AKS and Application Gateway Tutorial — Part 2

Rhodri Freer
6 min readSep 29, 2023

--

Introduction

In part 1 we created and tested our base cluster.

In part 2 we’ll add in application gateway and create an additional kubernetes manifest to test it. I’ll also show you some troubleshooting tips to verify the AGIC ingress pod is working as it should.

Full Code

For those of you who just want to see the final code; it’s here, part2 :-)

If you’re interested in how these lessons can be combined together in a more production ready pipeline then take a look at the demo on my github which builds on these concepts further; demo 1.

Adding Application Gateway To AKS

1. Assumptions

You’ve completed part 1, tested your deployment and have nginx running.

2. Updating the AKS Module

To add application gateway we need to add the following code.

# Install application gateway.
# Note: you MUST add a tweak to give the auto-created ingress uai the contributor iam role on the subnet or vnet; see tweak 1.
# Note: it's normal for the pod/ingress-appgw-deployment to restart / crashloopbackoff whilst it's deploying agic.
# Note: it can take ~15-20 mins for the agw to install; monitor pod/ingress-appgw-deployment logs for errors and progress.
ingress_application_gateway {
  subnet_id = var.subnet_appgw
}
# Required to set IAM role on appgw subnet.
output "aks_uai_appgw_object_id" { value = azurerm_kubernetes_cluster.kc.ingress_application_gateway[0].ingress_application_gateway_identity[0].object_id }
  • Update terraform/modules/az_aks/main.tf as below.
# Declare variables.
variable "location" {}
variable "resource_group_name" {}
variable "aks_name" {}
variable "subnet_aks" {}
variable "subnet_appgw" {}

# Create the aks cluster.
resource "azurerm_kubernetes_cluster" "kc" {
  location            = var.location
  resource_group_name = var.resource_group_name
  node_resource_group = "${var.resource_group_name}-nrg"
  name                = var.aks_name
  dns_prefix          = "${var.aks_name}-dns"

  # Set the default node pool config.
  default_node_pool {
    name           = "default"
    node_count     = "1"
    vm_size        = "Standard_B2s"
    vnet_subnet_id = var.subnet_aks # Nodes and pods will receive ip's from here.
  }

  # Set the identity profile.
  identity {
    type = "SystemAssigned"
  }

  # Set the network profile.
  network_profile {
    network_plugin = "azure"
  }

  # Install application gateway.
  # Note: you MUST add a tweak to give the auto-created ingress uai the contributor iam role on the subnet or vnet; see tweak 1.
  # Note: it's normal for the pod/ingress-appgw-deployment to restart / crashloopbackoff whilst it's deploying agic.
  # Note: it can take ~15-20 mins for the agw to install; monitor pod/ingress-appgw-deployment logs for errors and progress.
  ingress_application_gateway {
    subnet_id = var.subnet_appgw
  }

}

# Required for helm provider config.
output "aks_config" { value = azurerm_kubernetes_cluster.kc.kube_config }

# Required to set access policy on key vault.
output "aks_uai_agentpool_object_id" { value = azurerm_kubernetes_cluster.kc.kubelet_identity[0].object_id }

# Required when setting up csi driver secret provier class.
output "aks_uai_agentpool_client_id" { value = azurerm_kubernetes_cluster.kc.kubelet_identity[0].client_id }

# Required to set IAM role on appgw subnet.
output "aks_uai_appgw_object_id" { value = azurerm_kubernetes_cluster.kc.ingress_application_gateway[0].ingress_application_gateway_identity[0].object_id }

Notice the new output we’ve created also, which outputs the object id for the auto-created ingress user assigned identity. We’ll need this to add the ingress uai with the contributor role on the appgw vnet; which we’ll do shortly.

3. Updating the Outputs

You don’t need to do this, but I like to print my outputs to the screen to help with any verification or troubleshooting I may need to do.

  • Copy and paste the text below into terraform/99-outputs.tf.
# AKS
output "aks_uai_agentpool_object_id" { value = module.az_aks.aks_uai_agentpool_object_id }
output "aks_uai_agentpool_client_id" { value = module.az_aks.aks_uai_agentpool_client_id }
output "aks_uai_appgw_object_id" { value = module.az_aks.aks_uai_appgw_object_id }

4. AKS-Tweaks Config

Next we’ll populate a couple more files. These contain the configuration for any tweaks we’ll need to make to get things working. The first one being adding the auto-created ingress uai with the contributor role over the appgw vnet.

  • Update terraform/modules/az_aks_tweaks/main.tf as below.
# Tweak 1
variable "tw1_aks_vnet_id" {}
variable "tw1_aks_uai_appgw_object_id" {}
resource "azurerm_role_assignment" "contributor-to-aks-ingress-on-appgw-vnet" {
  scope                = var.tw1_aks_vnet_id
  role_definition_name = "Contributor"
  principal_id         = var.tw1_aks_uai_appgw_object_id
}
  • Update terraform/13-aks-tweaks.tf as below.
# Create kubernetes cluster tweaks.
module "az_aks_tweaks" {

  depends_on = [module.az_aks]           # Wait for dependencies.
  source     = "./modules/az_aks_tweaks" # The path to the module.

  # Tweak 1 - set 'contributor' role assignment for the auto-created 'ingressapplicationgateway' uai on the appgw vnet; otherwise it can't manage ingress.
  tw1_aks_vnet_id             = azurerm_virtual_network.vn.id         # The aks vnet id.
  tw1_aks_uai_appgw_object_id = module.az_aks.aks_uai_appgw_object_id # The client_id for the auto-created 'ingressapplicationgateway' uai .

}

5. Terraform Apply

At this point we’re ready to deploy our updated code.

# Ensure you are in the terraform directory.
terraform init -reconfigure -backend-config="key=env-prd.tfstate"
terraform validate
terraform apply -var-file="vars/global.tfvars" -var-file="vars/env-prd.tfvars"

Verifying Our Work

1. Ingress IAM Role Assignment

Within the Azure portal, pop over to the new AKS vnet and under Access Control (IAM) confirm the ingressapplicationgateway uai has been assigned the contributor role.

2. Monitoring Window Updates

If you have the luxury of more than one monitor, then I like to setup my second screen with a few windows. I use an app called Windows PowerToys, and a feature of that called FancyZones to speed up snapping my windows into place. The windows I use are:

  • The monitoring window we setup in part 1.
  • A separate PowerShell window for viewing logs; the logs window.
  • A web browser for testing.

NOTE: I’ll keep referring to the ‘monitoring window’ and ‘logs window’ from now on so it’s important that you have these running if you want to follow along. If you don’t have a second screen, then just have them running in the background so you can easily switch to them.

3. Application Gateway Deployment Progress

a. The Monitoring Window
In the monitoring window you should see a new pod resource that has popped up called something like pod/ingress-appgw-deployment. It could be in a CrashLoopBackoff state with a number of restarts showing. This is normal whilst it’s waiting for application gateway to install.

b. The Logs Window
In the logs window run the following command to view the ingress pods logs in real time. Don’t forget to replace the pod name below with the one from your monitoring window.

kubectl logs -f ingress-appgw-deployment-6bc4885f57-7tpm2 -n kube-system

The log should look something like below and may keep stopping. Just keep repeating the command above to restart it.
You’ll notice from the timestamp that it can take ~5–10 minutes to complete the installation.

Once complete it’ll scroll through a whole load of config on the screen and end with something like this. You need to wait for this to complete before continuing.

I like to leave this log running so that when we deploy anything that changes application gateway, we can see the deployment changes happening in real time.

c. Errors You Can Ignore

  • Error getting route table
    This is only relevant when using kubenet networking, but we’re using azure cni so it can be safely ignored.
  • ErrorMissingResourceGroup
    This is because we haven’t taken the default auto-created AKS resource group name. It can be safely ignored.

Updating Our Application

We’re now ready to update our application to enable access via application gateway.

1. Ingress (HTTP)

Create a new file; k8s/3-nginx-80-ingress.yaml, and copy and paste the text below into it.
This will create an ingress resource using application gateway over HTTP.

---
# INGRESS
# Browse to http://[public_ip]
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx
  namespace: titan
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
spec:
  rules:
  - http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: nginx
            port:
              number: 80
---

2. Deploy the Ingress (HTTP)

Run the command below to deploy the ingress. This assumes you’ve already run files 0, 1 and 2. If not run them first.

# Ensure you are in the k8s directory.
kubectl apply -f .\3-nginx-80-ingress.yaml

3. Monitoring

Things to look for …

  • Firstly when you run the command above you should see the logs window spring into life and start configuring the application gateway.
  • Secondly, if you keep an eye on the monitoring window you should see a new ingress resource pop up, with an external ip address. If you enter this into your browser as an http address it should bring up the nginx application; http://20.105.9.137.

Wrap Up

If you’ve made it this far and everything is working, congratulations.

In part 3 we’ll introduce a purchased TLS certificate so we can secure our application gateway traffic over HTTPS :-)

“He who laughs most, learns best.” — John Cleese

Kubernetes
Application Gateway
Terraform
Helm

--

--

Rhodri Freer

Written by Rhodri Freer

56 Followers

DevOps & Cloud Engineer

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams