How to Renew Let’s Encrypt Certificates Managed by cert-manager on Kubernetes

Gibbous moon in September 2020. Shot on Canon 5D Mark III, 200mm at f13 (EF70-200 f2.8L II USM). Photo by Dzero Labs.

So, you’ve set up TLS on your Kubernetes cluster managed by cert-manager, serving up Let’s Encrypt certificates. And then you get an email like this:

The dreaded certificate renewal email

If you’re like me and it’s your first experience with Let’s Encrypt on Kubernetes, maybe you’ve gone into a panic, thinking: “Now WHAT?”

Never fear! Renewing the certificate is actually pretty easy!

First things first. By default, Let’s Encrypt certificates expire every 90 days. Let’s Encrypt usually sends an e-mail (like the one above) to the address associated with the Certificate resource created in Kubernetes to remind the cluster admin to renew it. As a best practice, certificates should be renewed about 30 days before expiry.

NOTE: I’m assuming that you’re running Ambassador Edge Stack API Gateway on TLS using cert-manager to manage your Let’s Encrypt Certificate. The steps below are based on this setup. This means that things may be a bit different if you have a different setup and/or are renewing Certificates for another API Gateway or Ingress Controller.

1- Check certificate status

Before we begin, let’s first take a look at our certificate. To check certificate status, run the command below.

kubectl describe certificates ambassador-certs -n ambassador

My output looked like this:

The certificate is going to expire in March

Let’s take a look at what the highlighted fields mean:

• Not After is the Certificate’s expiry date. In our case, it’s 2021–03–12.
• Not Before is typically the date that the certificate was created. That is, if you didn’t explicitly populate it this field in your Certificate resource YAML.
• Renewal Time is the 30-day mark before the Certificate expires.

2- Delete the Certificate and Secret

In our case, we configured TLS for Ambassador, so we’re deleting both the Certificate resource and its accompanying Secret resource from the ambassador namespace in our cluster.

To check to see TLS is actually disabled, try going to your Ambassador URL using https. For example, https://<my_domain>.com

If you get something like the screen below, then good. That’s what we expected.

3- Re-create the certificate in Kubernetes

Okay, so now that we’ve nuked our Certificate and its accompanying Secret, we need to recreate it. Here’s the YAML that creates the Certificate resource, which we’re calling ambassador-certs:

Be sure to replace Line 16 with your Ambassador DNS name or FQDN. Note that IPs will not be accepted.

Now apply it to Kubernetes:

kubectl apply -f ambassador-certificate-definiton.yml

4- Verify

After we apply the above YAML to Kubernetes, it should create both a Certificate and a Secret, both of which are called ambassador-certs (because that’s how we named them in the YAML in Step 3 – see lines 4 and 10).

It might take a bit of time before the certificate is ready, so wait 30–40 seconds or so before running the command below:

kubectl describe certificates ambassador-certs -n ambassador

If all goes well, you should see something like this:

The new certificate expires in April

Let’s take a look at the highlighted fields:

• Reason is Ready (cert is ready)
• Status is True (cert hasn’t gone caca)
• Not After is now 2021–04–13. The old value was2021–03–12. Yay — it’s been updated!

NOTE: Yes, I probably renewed a little too early, since the only thing I managed to do was buy myself an extra month, but this was for demo purposes only.

Now that we’ve checked on the Certificate creation, let’s check on the ambassador-certs Secret as well. It should have been created automagically when we applied the cert-creation YAML in Step 3 above:

kubectl get secrets -n ambassador 

The output should look something like this:

The accompanying secret has been created successfully

Yes, we have a new secret called ambassador-certs, and it’s of type kubernetes.io/tls. Just as we expected. Success!

Keeping it DevOpsy

In order to keep with DevOps principles, we shouldn’t have to be running the above steps manually every time our certificate needs to be renewed. So let’s put it all together into a handy-dandy script:

End-to-end certificate renewal script

NOTE: If you’re using Bash instead of MacOS/BSD, your date command will be:date -d <date_string>. Check out this handy reference here.

Remember that Let’s Encrypt recommends that you renew your certificates 30 days before expiry. This minimizes service disruption and headaches. Ideally, you should create a cron job that runs every few days to check to see if today’s date is on or before the date on the certificate’s Renewal Time field. If it is, then run the above script to regenerate the certificate.

We can get Renewal Time like this:

kubectl get certificate -n ambassador -o=jsonpath='{.items[0].status.renewalTime}'

Final Thoughts

We learned today that it’s not terribly complicated to renew Let’s Encrypt Certificates managed by cert-manager.

I recognize that the approach taken today is a bit of a layperson’s approach, but it works! Remember that the best way to keep certificate renewal from creeping up on you is to run a cron job that checks the certificate’s Renewal Time before running that handy all-in-one script above.

If you have thoughts on how to improve this process and/or the above script, please share in the comments below!

Image source here

Peace, love, and code.

References

• Certificate Resource Overview (cert-manager)
• Certificate API Documentation (cert-manager)
• Overview of cert-manager and Let’s Encrypt

Other stories in my ArgoCD journey

Want to know more on how we got here? Check out the other stories in my ArgoCD journey below!

Installing Ambassador, ArgoCD, and Tekton on Kubernetes

Using Tekton and ArgoCD to Set Up a Kubernetes-Native Build & Release Pipeline

Configuring SSO with Azure Active Directory on ArgoCD

Turbocharge ArgoCD with App of Apps Pattern and Kustomized Helm