Open in app

Sign in

Write

Sign in

A Terraform, AKS and Application Gateway Tutorial — Part 4

Rhodri Freer
8 min readSep 29, 2023

--

Introduction

In part 3 we added a purchased SSL certificate to our environment, and configured our application to work via application gateway over HTTPS.

In part 4 we’ll introduce a free certificate provider, Cert-Manager and modify our application to use that certificate instead.

Full Code

For those of you who just want to see the final code; it’s here, part4 :-)

If you’re interested in how these lessons can be combined together in a more production ready pipeline then take a look at the demo on my github which builds on these concepts further; demo 1.

Adding TLS To Application Gateway Using Cert-Manager

1. Assumptions

You’ve completed parts 1 and 2, tested your deployment and have nginx accessible over HTTP via application gateway.

2. Update Terraform Files

Cert-Manager is a certificate controller for Kubernetes. It will obtain certificates from a variety of issuers and ensure the certificates are valid and up-to-date, and will attempt to renew certificates at a configured time before expiry. For more details see https://cert-manager.io.

We’ll install Cert-Manager using a helm chart by updating the following files. If you did part 3, we don’t need akv2k8s any more, so that config can come out.

  • terraform/modules/az_helm/main.tf
# Install cert-manager.
resource "helm_release" "cert-manager" {
  name             = "cert-manager"
  chart            = "cert-manager"
  version          = "1.12.3"
  repository       = "https://charts.jetstack.io"
  namespace        = "cert-manager"
  atomic           = true
  create_namespace = true
  set {
    name  = "installCRDs"
    value = "true"
  }
}

3. Terraform Apply

At this point we’re ready to deploy our updated code.

# Ensure you are in the terraform directory.
terraform init -reconfigure -backend-config="key=env-prd.tfstate"
terraform validate
terraform apply -var-file="vars/global.tfvars" -var-file="vars/env-prd.tfvars"

4. Verifying Our Update

If you keep an eye on the monitoring window, you’ll see a number of resources pop up in the cert-manager namespace.

Updating Our Application

We’re now ready to update our application to enable access via application gateway over HTTPS, using Cert-Manager as the certificate provider.

1. ClusterIssuer

The first thing we need to configure after installing cert-manager is a ClusterIssuer. This is a resource that represents a certificate authority (CA) able to sign certificates in response to certificate signing requests. For more details see https://cert-manager.io/docs/configuration.

Create a new file; k8s/6-cluster-issuer.yaml, and copy and paste the text below into it.

Note the ‘server: https://acme-staging-v02.api.letsencrypt.org/directory’ line. This is saying, whilst we are playing around let’s just use the ACME staging server as the live one limits issuing certificates to 50 per week. We’ll update this later.

Update the # UPDATE HERE lines.

---
# CLUSTERISSUER
# kubectl get clusterissuer      # Check if the clusterissuer has been created and ready status is true.
# kubectl describe clusterissuer # Check if the clusterissuer has any error events logged.
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-issuer
spec:
  acme:
    email: rhod3rz@outlook.com                                        # UPDATE HERE < email to contact you about expiring certificates & issues.
    server: https://acme-staging-v02.api.letsencrypt.org/directory    # < use this staging issuer when testing to avoid hitting rate limits on prod (50 per week).
    # server: https://acme-v02.api.letsencrypt.org/directory          # < use this prod issuer when ready to go live.
    privateKeySecretRef:
      name: letsencrypt-issuer                                        # < secret resource used to store the account's private key.
    solvers:
    - http01:
        ingress:
          class: azure/application-gateway
---

2. Ingress (HTTPS with Cert-Manager)

Create a new file; k8s/7-nginx-443-ingress-cert-manager.yaml, and copy and paste the text below into it.
This will create an ingress resource using application gateway and Cert-Manager over HTTPS.

Update the # UPDATE HERE lines.

---
# INGRESS
# Ensure a DNS A record mapping exists for your URL; Cert-Manager will use this to verify you own the domain.
# kubectl get secret -n titan                      # Confirm the secret has been created.
# kubectl describe secret prd.rhod3rz.com -n titan # View the secret details.
# kubectl get certificate -n titan                 # Check if a cert has been created and the ready status is true.
# kubectl describe certificate -n titan            # If the ready status is false; run this and look at the events.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx
  namespace: titan
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway # < add annotation indicating the ingress to use.
    appgw.ingress.kubernetes.io/ssl-redirect: "true"       # < add annotation to redirect 80 requests to 443.
    cert-manager.io/cluster-issuer: letsencrypt-issuer     # < add annotation indicating the cert issuer to use.
spec:
  tls: # < placing a host in the TLS config will determine what ends up in the cert's subjectAltNames.
  - hosts:
    - prd.rhod3rz.com # UPDATE HERE
    secretName: prd.rhod3rz.com # UPDATE HERE < cert-manager will store the created certificate in this secret.
  rules:
  - host: prd.rhod3rz.com # UPDATE HERE
    http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: nginx
            port:
              number: 80
---

3. Deploy the Ingress (HTTPS with Cert-Manager)

Before you run the deployment below, we’re going to do something a little odd; if you already have a DNS A record setup that’s pointing to the puplic ip address of the application gateway, then delete it. We’ll add it back later, but it helps to understand the Cert-Manager process if it doesn’t work first time.

Run the commands below to deploy the ingress.

We’ll wipe out what’s there first and deploy fresh. Run each line one at a time and watch your monitoring and logs windows to see the changes taking place.

# Ensure you are in the k8s directory.
kubectl delete namespace titan
kubectl apply -f .\0-namespace.yaml
kubectl apply -f .\1-nginx-80-pod.yaml
kubectl apply -f .\2-nginx-80-service.yaml
kubectl apply -f .\6-cluster-issuer.yaml
kubectl apply -f .\7-nginx-443-ingress-cert-manager.yaml

4. Troubleshooting Cert-Manager

If you keep an eye on the monitoring window you should see your resources pop up, but also a few that have the name ‘acme-http-solver’ in them. These are resources Cert-Manager spins up to verify your domain name. In my case it’s trying to find an A record for prd.rhod3rz.com > 51.104.178.170, but as that doesn’t exist it’s failing. Once it succeeds those resources will disappear.

Let’s dig a little deeper to see what’s going on.

  • Check if the clusterissuer has been created and the ready status is true.
kubectl get clusterissuer

NAME                 READY   AGE
letsencrypt-issuer   True    26m
  • Check if the clusterissuer has any error events logged.
kubectl describe clusterissuer
  • Confirm the secret has been created.
kubectl get secret -n titan

NAME                    TYPE     DATA   AGE
prd.rhod3rz.com-nqngs   Opaque   1      31m
  • View the secret details. If this were successful you’d see a tls.crt listed also.
kubectl describe secret prd.rhod3rz.com -n titan

Name:         prd.rhod3rz.com-nqngs
Namespace:    titan
Labels:       cert-manager.io/next-private-key=true
              controller.cert-manager.io/fao=true
Annotations:  <none>
Type:  Opaque
Data
====
tls.key:  1704 bytes
  • Check if a cert has been created and the ready status is true.
kubectl get certificate -n titan

NAME              READY   SECRET            AGE
prd.rhod3rz.com   False   prd.rhod3rz.com   35m
  • If the ready status is false; run this and look at the events. Don’t forget we’re not expecting it to work just yet as we haven’t updated DNS.
kubectl describe certificate -n titan

5. Update DNS

Wherever you’re DNS is hosted, it’s now time to update it. Create an A record mapping the public ip of the application gateway to your domain / certificate name.

If you keep an eye on the monitoring window you should see the acme-http-solver resources disappear as they succeed.

If you keep an eye on the logs window you should see the configuration getting pushed to the application gateway.

To verify this is now working rerun the commands below.

  • View the secret details. You’ll see a tls.crt is now listed.
kubectl describe secret prd.rhod3rz.com -n titan
Name:         prd.rhod3rz.com
Namespace:    titan
Labels:       controller.cert-manager.io/fao=true
Annotations:  cert-manager.io/alt-names: prd.rhod3rz.com
              cert-manager.io/certificate-name: prd.rhod3rz.com
              cert-manager.io/common-name: prd.rhod3rz.com
              cert-manager.io/ip-sans:
              cert-manager.io/issuer-group: cert-manager.io
              cert-manager.io/issuer-kind: ClusterIssuer
              cert-manager.io/issuer-name: letsencrypt-issuer
              cert-manager.io/uri-sans:
Type:  kubernetes.io/tls
Data
====
tls.crt:  5656 bytes
tls.key:  1675 bytes
  • Check the ready status has changed to true.
kubectl get certificate -n titan

NAME              READY   SECRET            AGE
prd.rhod3rz.com   True    prd.rhod3rz.com   49m
  • Verify the certificate has been issued.
kubectl describe certificate -n titan

Type    Reason     Age    From                                       Message
----    ------     ----   ----                                       -------
Normal  Issuing    49m    cert-manager-certificates-trigger          Issuing certificate as Secret does not exist
Normal  Generated  49m    cert-manager-certificates-key-manager      Stored new private key in temporary Secret resource "prd.rhod3rz.com-nqngs"
Normal  Requested  49m    cert-manager-certificates-request-manager  Created new CertificateRequest resource "prd.rhod3rz.com-bchm4"
Normal  Issuing    9m42s  cert-manager-certificates-issuing          The certificate has been successfully issued

6. The Moment of Truth

You are now ready to test, browse to your URL.

Hmm, was this what you were expecting?

It’s actually a good sign. If you drill into the certificate details you’ll see it’s using the staging ACME issuer, which isn’t a trusted root on your device; hence the ‘Certificate is not valid’ message. If you remember in an earlier step when we created the ClusterIssuer, we told it to use the staging server.

Now we’ve confirmed all is working, you can follow the steps below to switch over to the production ACME issuer.

  • Delete the namespace and the clusterissuer.
kubectl delete namespace titan
kubectl delete clusterissuer letsencrypt-issuer
  • Update k8s/6-cluster-issuer.yaml as below. Remember to enter your email address.
---
# CLUSTERISSUER
# kubectl get clusterissuer      # Check if the clusterissuer has been created and ready status is true.
# kubectl describe clusterissuer # Check if the clusterissuer has any error events logged.
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-issuer
spec:
  acme:
    email: rhod3rz@outlook.com                                        # UPDATE HERE < email to contact you about expiring certificates & issues.
    # server: https://acme-staging-v02.api.letsencrypt.org/directory  # < use this staging issuer when testing to avoid hitting rate limits on prod (50 per week).
    server: https://acme-v02.api.letsencrypt.org/directory            # < use this prod issuer when ready to go live.
    privateKeySecretRef:
      name: letsencrypt-issuer                                        # < secret resource used to store the account's private key.
    solvers:
    - http01:
        ingress:
          class: azure/application-gateway
---
  • Redeploy the application.
# Ensure you are in the k8s directory.
kubectl apply -f .\0-namespace.yaml
kubectl apply -f .\1-nginx-80-pod.yaml
kubectl apply -f .\2-nginx-80-service.yaml
kubectl apply -f .\6-cluster-issuer.yaml
kubectl apply -f .\7-nginx-443-ingress-cert-manager.yaml

If you’re quick you should see the acme-http-solver resources pop up again, but this time as the DNS record exists everything should just work. If you check the certificate on the website now you’ll see it’s using a trusted certificate issued by Let’s Encrypt.

Wrap Up

If you’ve made it this far and everything is working, congratulations. In part 5 we’ll introduce the Secret Store CSI Driver which allows us to inject secrets from key vault into our pods :-)

“The more that you read, the more things you will know. The more that you learn, the more places you’ll go.” — Dr. Seuss

Kubernetes
Application Gateway
Terraform
Helm

--

--

Rhodri Freer

Written by Rhodri Freer

60 Followers

DevOps & Cloud Engineer

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams