Public TLS Certificate Automation in LKE using Certbot & KIC

Certificates on the Internet are a very important part of everyday life in this day and age. When certificates stop working, and my browser can not authenticate that you are who you say you are, all Hell tends to break loose. And probably for the best. This keeps us from giving sensitive information to people who should not have it. But from a backend administration perspective, certificates can be a real pain in the backend — pun intended. In this post, I am going to outline a great process that I like to use to automate this certificate handling through the use of a special container, a configmap, Kubernetes secrets and my good friend Certbot. And best of all, this process is FREE and makes use of FREE TLS certificates validated by a public Certificate Authority. I will also show you how to use this with Kong Ingress Controller for Kubernetes (in Akamai Connected Cloud using the Linode Kubernetes Engine). If you would like to build along and do not already have a Akamai Connected Cloud account — Sign up here for $100 free credit. You will need a registered domain name to build along.

Getting Started

I am going to assume that you have the basic knowledge to deploy an LKE cluster in the Akamai Connected Cloud. As well, I am going to assume you have kubectl and helm on your local machine (or a bastion you use). If you have a Docker environment at your disposal, feel free to steal my Docker files and make your own container to store in your Docker Hub repo and use with the Kubernetes manifests provided — otherwise just skip step 2.

Step 1 — Prepare your environment.

Clone the repo! This will make everything easier. git clone https://github.com/eilerb1011/k8s-autocert

If you do not already have an LKE Cluster, create one and download the kubeconfig to your working directory. You only need the smallest cluster possible to complete this — a single 1x2 node on a non-HA LKE cluster will suffice.

Step 2 — Build theContainer (You can skip this)

Using the Docker folder in the repo, build your container using your Docker environment. Unless you want to make some changes to the container, you can leave everything as is — just be sure to change the below commands to your Dockerhub username and the container name you want to use.

docker build -t myDockeruser/my-cool-certbot:latest .
docker login
docker push myDockeruser/my-cool-certbot:latest

Step 3— Prepare the Cluster

Now that you have the repo and your cluster, you will need to do a few things to prep by creating a secret that contains the kubeconfig as well as a configmap that contains information needed by Certbot to create the TLS cert. For the purpose of this exercise, we are going to create a secret from the default kubeconfig file — However, it is recommended to lock things down by creating a role that will only be able to delete and create certs through the use of roles, bindings and a custom kubeconfig.

Last, in this exercise we will use Kong Ingress Controller to direct traffic inbound from the Internet to our intermittent Certbot container. This will allow us to keep the webroot open for other services, while creating a certificate for a hostname that can be applied to many different webservices in the LKE cluster. So we will install Kong and apply a basic ingress that sends traffic to certbot as well as sends traffic from our webroot to a special container that echos back a node name. And once the Kong Ingress is in place, we will need to tie the fully qualified domain name, as we have documented in our configmap, to the IP Address of the Kong Ingress in our authoritative DNS.

Before creating the configmap, decide on the hostname in your domain that will be used for your cluster. Mine will be myhost.linode.eiler.rocks. Edit the config.yaml provided in the repo and change the clusterfqdn variable to match your chosen name. Then edit the email address to match the email address you want to be notified on with certificate issues. Save the file. Now we can deploy the secret, the configmap, install Kong (if not already installed) and create the DNS entry.

kubectl create secret generic kcon --from-file=config=kubeconfig.yaml
kubectl apply -f config.yaml
#If Kong is already installed, skip this step
helm repo add kong https://charts.konghq.com
helm repo update
helm install kong/kong --generate-name --set ingressController.installCRDs=false --create-namespace --namespace kong

In the commands above, replace kubeconfig.yaml with the name of your own kubeconfig. And now you should have a Kong ingress running in the Kong namespace. You should also have your kubeconfig exposed as a secret named kcon. And you should have a configmap named cert-vars that holds the variables for our Certbot process. The next step is to retrieve your IP Address using kubectl get svc -n kong. This will output something similar to below:

kubectl output

Use the External IP listed in the kong-proxy line and create an A record in your authoritative DNS server using the host name from the fqdn you have decided to use. **You must do this or else your TLS cert will not be issued.

Step 4 — Deploy Containers & Services

The next step will deploy 2 services and a container. The 2 services will allow for ingress from the Internet and direct traffic to our pods — both for Certbot and for NGINX as our main web service.

kubectl apply -f certbot-svc.yaml
kubectl apply -f nginx.yaml

Now run kubectl get pods and kubectl get svc to make sure you have an NGINX deployment running as well as a service for each NGINX and Certbot

kubectl get svc

Step 5 — Configure Ingress

Before we unleash Certbot to give us the authority to say we are who we say we are, we need to let Kong know where everything is at. In in the ingress.yaml from the repo, replace both entries replace.with.your.fqdn with your actual fqdn as listed in your authoritative DNS. Everything else can be left alone.

But, to provide insight on the rest of the ingress configuration. These entries tell Kong which hosts to accept on which services. Here we allow our fqdn on HTTP and HTTPS. The secret name is the name of the Kubernetes secret where we will store our TLS certificate. (This happens automagically using the certbot container.) The rules simply provide Kong a mechanism to direct traffic. The magic here is that we direct /.well-known/acme-challenge to the Kubernetes service named certbot. This allows for the certbot challenge-repsponse mechanism to work without disrupting traffic on /, which is mapped to the Kubernets service nginx.

Step 6 — Apply Cert Creation Jobs & Test

The last step will be to deploy Certbot itself. If you have built your own container — change the image in the cert-cron.yaml and cert-job.yaml files to reflect your repo and image. Else, leave them alone and deploy:

kubectl apply -f cert-job.yaml
kubectl apply -f cert-cron.yaml

This will deploy the certbot container that we built in step 2. This container, stands up an Apache server to accept the Certbot challenge-response. It then runs Certbot in silent mode using the configmap created in our preparation. Last, it takes the created TLS cert and stores it in the Kubernetes secret, sitecert, leveraging the config stored in the kcon secret. And then Kong references sitecert to apply TLS to connections.

Now, run kubectl get secrets --watch

When you see the new secret named sitecert pop up, open a web browser and go to https://your.fqdn. You should see a page with the node name that is running NGINX. And that is it. Your cron job will run every 30 days, create a new cert for your fqdn and place it in the sitecert secret pulled by Kong.

Response from NGINX

You will now have automagic TLS certs for any services on paths in your main fqdn. Enjoy!