A Terraform, AKS and Application Gateway Tutorial — Part 3
Introduction
In part 2 we added application gateway to our cluster and created an ingress resource to use application gateway; albeit over HTTP only.
In part 3 we’ll introduce a key vault to store certificates and secrets and update our application to be accessed via application gateway over HTTPS.
Part 3 focuses on organisations that want to use their own SSL certificates to secure application gateway. You’ll need your own registered domain name and an SSL certificate, or be happy to buy them; they are relatively cheap.
If you’re after a free SSL certificate option then I’ll cover Cert-Manager in part 4. If that’s the case then still follow along here to complete the key vault creation as we’ll need it for ‘Part 5 — Adding The CSI Driver To Import Secrets Into Our Pods’.
Full Code
For those of you who just want to see the final code; it’s here, part3 :-)
If you’re interested in how these lessons can be combined together in a more production ready pipeline then take a look at the demo on my github which builds on these concepts further; demo 1.
Adding TLS To Application Gateway Using A Purchased Certificate
We’ll begin by creating a key vault to store our certificate. We’ll also add a secret we can use in part 5 when we introduce the secret store CSI driver.
When you create a key vault, by default nobody has access to view secrets , certificates etc. The config below will also update access policies to allow access to your admin account, the Terraform service principal and the aks agentpool uai.
1. Assumptions
You’ve completed parts 1 and 2, tested your deployment and have nginx accessible over HTTP via application gateway.
2. Key Vault Config
To add key vault we need to edit & populate a few more files.
- terraform/modules/az_keyvault/main.tf
# Declare variables.
variable "location" {}
variable "resource_group_name" {}
variable "keyvault_name" {}
# Get the terraform service principal config.
data "azurerm_client_config" "current" {}
# Create a keyvault.
resource "azurerm_key_vault" "kv" {
name = var.keyvault_name
location = var.location
resource_group_name = var.resource_group_name
tenant_id = data.azurerm_client_config.current.tenant_id
sku_name = "standard"
soft_delete_retention_days = "7"
purge_protection_enabled = false
}
# Required to set keyvault access policies.
output "keyvault_id" { value = azurerm_key_vault.kv.id }- terraform/14-keyvault.tf
# Create keyvault.
module "az_keyvault" {
source = "./modules/az_keyvault" # The path to the module.
location = var.location # The location.
resource_group_name = azurerm_resource_group.rg.name # The resource group.
keyvault_name = "kv-${var.environment}-${var.application_code}-${var.unique_id}" # Keyvault name.
}
# Create keyvault access policies for the aks agentpool uai.
resource "azurerm_key_vault_access_policy" "kvap_agentpool" {
depends_on = [module.az_keyvault]
key_vault_id = module.az_keyvault.keyvault_id
tenant_id = data.azurerm_client_config.current.tenant_id
object_id = module.az_aks.aks_uai_agentpool_object_id
key_permissions = ["Get"]
secret_permissions = ["Get"]
certificate_permissions = ["Get"]
}
# Create keyvault access policies for your user account and the terraform service principal.
resource "azurerm_key_vault_access_policy" "kvap_admin_users" {
depends_on = [module.az_keyvault]
for_each = { for my_kv in var.keyvault_policies : my_kv.access_policy_name => my_kv }
key_vault_id = module.az_keyvault.keyvault_id
tenant_id = data.azurerm_client_config.current.tenant_id
object_id = each.value.object_id
key_permissions = try(each.value.key_permissions, [])
secret_permissions = try(each.value.secret_permissions, [])
certificate_permissions = try(each.value.certificate_permissions, [])
}
# Create a sample secret so we have something to retrieve later.
resource "azurerm_key_vault_secret" "kvs_username" {
depends_on = [module.az_keyvault, resource.azurerm_key_vault_access_policy.kvap_admin_users]
name = "titan-username"
value = "bart_simpson"
key_vault_id = module.az_keyvault.keyvault_id
}
# Create a sample secret so we have something to retrieve later.
resource "azurerm_key_vault_secret" "kvs_password" {
depends_on = [module.az_keyvault, resource.azurerm_key_vault_access_policy.kvap_admin_users]
name = "titan-password"
value = "C0ngr4tul4t10n5"
key_vault_id = module.az_keyvault.keyvault_id
}- terraform/vars/global.tfvars
Update global.tfvars to add the section below.
Update the access_policy_name and object_id to reflect your admin user account, and the terraform service principal.
#================================================================================================
# Key Vault Policies
#================================================================================================
keyvault_policies = [
{
access_policy_name = "rhodri.freer" # UPDATE HERE.
object_id = "72c94293-5323-480c-86e0-097289139d2f" # UPDATE HERE.
key_permissions = [
"Backup", "Create", "Decrypt", "Delete", "Encrypt", "Get", "Import", "List", "Purge", "Recover", "Restore", "Sign", "UnwrapKey", "Update", "Verify", "WrapKey", "Release", "Rotate", "GetRotationPolicy", "SetRotationPolicy"
]
secret_permissions = [
"Backup", "Delete", "Get", "List", "Purge", "Recover", "Restore", "Set"
]
certificate_permissions = [
"Backup", "Create", "Delete", "DeleteIssuers", "Get", "GetIssuers", "Import", "List", "ListIssuers", "ManageContacts", "ManageIssuers", "Purge", "Recover", "Restore", "SetIssuers", "Update"
]
},
{
access_policy_name = "sp-terraform-deployment" # UPDATE HERE.
object_id = "89b77e65-0434-4628-87db-9f1dd6c262d4" # UPDATE HERE.
key_permissions = [
"Backup", "Create", "Decrypt", "Delete", "Encrypt", "Get", "Import", "List", "Purge", "Recover", "Restore", "Sign", "UnwrapKey", "Update", "Verify", "WrapKey", "Release", "Rotate", "GetRotationPolicy", "SetRotationPolicy"
]
secret_permissions = [
"Backup", "Delete", "Get", "List", "Purge", "Recover", "Restore", "Set"
]
certificate_permissions = [
"Backup", "Create", "Delete", "DeleteIssuers", "Get", "GetIssuers", "Import", "List", "ListIssuers", "ManageContacts", "ManageIssuers", "Purge", "Recover", "Restore", "SetIssuers", "Update"
]
}
]- terraform/01-variables.tf
Update 01-variables.tf to include the key vault policies entry below.
#================================================================================================
# Environment Vars - Global
#================================================================================================
variable "tenant_id" {} # The azure tenant id.
variable "terraform_sp" {} # The terraform service principal.
variable "location" {} # Azure location to deploy resources in.
variable "application_code" {} # Project code used for naming.
variable "unique_id" {} # A unique id for naming.
#================================================================================================
# Environment Vars
#================================================================================================
variable "environment" {} # The environment name.
variable "subscription_id" {} # The subscription id.
#================================================================================================
# Key Vault Policies
#================================================================================================
variable "keyvault_policies" {} # Key vault policies.3. Terraform Apply
At this point we’re ready to deploy our updated code.
# Ensure you are in the terraform directory.
terraform init -reconfigure -backend-config="key=env-prd.tfstate"
terraform validate
terraform apply -var-file="vars/global.tfvars" -var-file="vars/env-prd.tfvars"Verify your code has performed the following steps:
- Deployed a new key vault
- Added a secret called ‘titan-password’
- Added a secret called ‘titan-username’
- Added three access policies
Adding an SSL Certificate to Key Vault
If you only want the free Cert-Manager option, then you can skip to part 4 now. If not then read on …
This is where things get a little tricky as purchasing and creating certificates is a topic in it’s own right. I’ll try and write an article on that in the future, but for now you’ll have to do some of the leg work yourself with loose guidance from me.
1. Purchase a Domain Name
I used GoDaddy and bought rhod3rz.com for less than $10/year.
I also set up rhod3rz.com as a DNS zone within my Azure subscription, and updated the GoDaddy nameservers to redirect there so I can do all my DNS record updates in Azure.
2. Purchase an SSL Certificate
I used Comodo SSL Store and bought a Comodo Positive Multi Domain SSL for about $30/year, https://comodosslstore.com/positive-multidomain-ssl.aspx. I chose this option as I wanted to register the san names of rhod3rz.com, dev.rhodr3rz.com, stg.rhodr3rz.com and prd.rhodr3rz.com.
If you just want a single name registered (e.g. test.mydomain.com), which would suffice for this tutorial then you could chose this option, https://comodosslstore.com/positivessl.aspx which is about $10/year.
Once you complete the verification process you’ll be able to log into https://certpanel.com and download your certificate.
Once downloaded you’ll get a .zip file with a load of files in it.
- Rename My_CA_Bundle.ca-bundle > mycert.crt
- Copy the contents of 1708667777.crt to the top of mycert.crt. This is now your certificate. You don’t need anything else.
3. Convert Certificate to PFX Format
In order to store a certificate in key vault, it must be in a .pfx format. In order to do that you’ll need access to the openssl command. I use WSL in Windows and fire up an ubuntu bash shell which has openssl installed out of the box.
You’ll need to run the command below to create the .pfx file where:
-out = the name of the file you want to output
-inkey = the private key you generated when creating the csr to submit to Comodo to create your certificate
-in = the file you just created above
openssl pkcs12 -export -out mycert.pfx -inkey mycert.key -in mycert.crt4. Upload Certificate to Key Vault
Now you have a .pfx file, pop over to the Azure portal, find the key vault and import the certificate; call it titan-certificate. If all has worked you’ll be able to click into the certificate and view it’s details.
Installing Azure Key Vault to Kubernetes (AKV2K8S)
1. Update / Create Terraform Files
AKV2K8S makes certificates available in AKS in a simple and secure way. For more details see https://akv2k8s.io.
We’ll install AKV2K8S using a helm chart by updating the following files.
- terraform/modules/az_helm/main.tf
# Install akv2k8s.
resource "helm_release" "akv2k8s" {
name = "akv2k8s"
chart = "akv2k8s"
version = "2.5.0"
repository = "https://charts.spvapi.no"
namespace = "akv2k8s"
atomic = true
create_namespace = true
}- terraform/15-helm.tf
# Install helm charts.
module "az_helm" {
depends_on = [module.az_aks] # Wait for dependencies.
source = "./modules/az_helm" # The path to the module.
}2. Terraform Apply
At this point we’re ready to deploy our updated code.
# Ensure you are in the terraform directory.
terraform init -reconfigure -backend-config="key=env-prd.tfstate"
terraform validate
terraform apply -var-file="vars/global.tfvars" -var-file="vars/env-prd.tfvars"3. Verifying Our Update
If you keep an eye on the monitoring window, you’ll see a number of resources pop up in the akv2k8s namespace.
Updating Our Application
We’re now ready to update our application to enable access via application gateway over HTTPS.
1. AzureKeyVaultSecret
Create a new file; k8s/4-akv2k8s.yaml, and copy and paste the text below into it.
This will read key vault and import the certificate and store it as a kubernetes secret.
---
# AZURE KEY VAULT SECRET
# kubectl get AzureKeyVaultSecret -n titan
# kubectl describe AzureKeyVaultSecret akvs-certificate-sync -n titan
# kubectl get secret -n titan
# kubectl describe secret titan-certificate -n titan
# kubectl get secret titan-certificate -o yaml -n titan
# kubectl logs deployment/akv2k8s-controller -n akv2k8s
apiVersion: spv.no/v2beta1
kind: AzureKeyVaultSecret
metadata:
name: akvs-certificate-sync
namespace: titan # The namespace to create the secret in. Secrets must be in the same namespace as the resource!
spec:
vault:
name: kv-prd-titan-230101 # Name of key vault.
object:
name: titan-certificate # Name of the certificate.
type: certificate
output:
secret:
name: titan-certificate # Kubernetes secret name.
type: kubernetes.io/tls # Kubernetes secret type.
---2. Ingress (HTTPS)
Create a new file; k8s/5-nginx-443-ingress.yaml, and copy and paste the text below into it.
This will create an ingress resource using application gateway over HTTPS.
Update the # UPDATE HERE lines.
---
# INGRESS
# Browse to https://prd.rhod3rz.com; you will need to update DNS or fudge your hosts file first!
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx
namespace: titan
annotations:
kubernetes.io/ingress.class: azure/application-gateway # < add annotation indicating the ingress to use.
appgw.ingress.kubernetes.io/ssl-redirect: "true" # < add annotation to redirect 80 requests to 443.
cert-manager.io/cluster-issuer: letsencrypt-issuer # < add annotation indicating the cert issuer to use.
spec:
tls: # < placing a host in the TLS config will determine what ends up in the cert's subjectAltNames.
- hosts:
- prd.rhod3rz.com # UPDATE HERE
secretName: prd.rhod3rz.com # UPDATE HERE < cert-manager will store the created certificate in this secret.
rules:
- host: prd.rhod3rz.com # UPDATE HERE
http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: nginx
port:
number: 80
---3. Deploy the Ingress (HTTPS)
Run the commands below to deploy the ingress.
We’ll wipe out what’s there first and deploy fresh. Run each line one at a time and watch your monitoring and logs windows to see the changes taking place.
# Ensure you are in the k8s directory.
kubectl delete namespace titan
kubectl apply -f .\0-namespace.yaml
kubectl apply -f .\1-nginx-80-pod.yaml
kubectl apply -f .\2-nginx-80-service.yaml
kubectl apply -f .\4-akv2k8s.yaml
kubectl apply -f .\5-nginx-443-ingress.yamlIf you need to troubleshoot the certificate or see what 4-akv2k8s.yaml did, then explore the following commands.
kubectl get AzureKeyVaultSecret -n titan
kubectl describe AzureKeyVaultSecret akvs-certificate-sync -n titan
kubectl get secret -n titan
kubectl describe secret titan-certificate -n titan
kubectl get secret titan-certificate -o yaml -n titan
kubectl logs deployment/akv2k8s-controller -n akv2k8s4. The Moment of Truth
If all went well you should have seen the logs window busy configuring application gateway and a new ingress resource pop up with a host name and public ip address in the monitoring window.
The last thing to do is update / create a DNS a record for the ip to host name mapping. Either that or fudge your hosts file locally to test.
Wrap Up
If you’ve made it this far and everything is working, congratulations.
In part 4 we’ll introduce a free certificate service going by the name Cert-Manager :-)
“The beautiful thing about learning is nobody can take it away from you.” — B.B. King
