Automated HTTPS with Let’s Encrypt and kube-lego

SSL is pretty important. Securing traffic ensures all data remains private, increases user trust, and reduces the risk of your application or user data being hijacked or otherwise screwed with. A green padlock in your address bar is pretty much table stakes for a web site. All you need is a certificate.

However…


The problem

Managing SSL certificates sucks. Getting hold of an SSL certificate can be a painful process. At its simplest, you need to:

  • Generate a Certificate Signing Request (CSR)
  • Supply that to a third party Certificate Authority (CA)
  • Prove ownership or control of the domain in question
  • Wait for the CA to provide the certificate
  • Install the certificate on your server
  • Configure your application to use the certificate.

Depending on the expiry date of the certificate, you’ll also need to renew it on a regular basis.

Oh, and the CA will often charge a lot for each new certificate, and for each renewal.

In larger organisations, this process is further complicated by the fact that the people who need the certificate — a development team, for example — likely won’t be the ones who deal with the CA, so now you have to generate the CSR, pass that on to an infrastructure team who will request the certificate from the CA, wait for the certificate to come back, then install it in the right place.

For a single application, that’s a cumbersome process. For an organisation with lots of applications, each with their own certificates with their own renewal dates, it’s an almost unmanageable amount of work. Throw in non-production environments for those applications, and you’re looking at major amounts of cost, time and effort to achieve something that shouldn’t be that hard.

This manual process also relies on both the development and infrastructure teams having the knowledge to carry out their tasks, and the Ops teams knowing when certificate renewal is required. Things can and do go wrong. A CSR might be incorrectly generated by the development team, meaning back-and-forth between the devs and the infrastructure team or the infrastructure and the CA; an Ops team might miss the renewal for a certificate, resulting in your application being insecure or just plain broken.

That’s not to say there aren’t ways round these issues. You could use a self-signed certificate that you create yourself without going through a CA, but these aren’t trusted by browsers — the best case scenario is you’ll get a security error when a user visits your site; the worst case is that user or company settings might prevent your site being accessible at all. Of course, you could also forgo HTTPS all together, but that’s a cop-out.

The good news is that none of the issues associated with SSL certificate generation and management are insurmountable.


Enter ACME

A couple of years ago, the Internet Security Research Group (ISRG), which is backed by companies such as the Electronic Frontier Foundation and Mozilla, released the first draft of the Automatic Certificate Management Environment (a backronym if ever there was one) specification. The spec comes from the following position:

With the exception of the CSR itself and the certificates that are issued, [the steps involved in certificate generation] are all completely ad hoc procedures and are accomplished by getting the human user to follow interactive natural-language instructions from the CA rather than by machine-implemented published protocols. In many cases, the instructions are difficult to follow and cause significant confusion. Informal usability tests by the authors indicate that webmasters often need 1–3 hours to obtain and install a certificate for a domain. Even in the best case, the lack of published, standardized mechanisms presents an obstacle to the wide deployment of HTTPS and other PKIX-dependent systems because it inhibits mechanization of tasks related to certificate issuance, deployment, and revocation.

If you (justifiably) have no desire to trawl through the entire spec, the aim is neatly summarised in the intro:

Use of this protocol should radically simplify the deployment of HTTPS. It would be nearly as easy to deploy with a CA-issued certificate as with a self-signed certificate. Furthermore, the maintenance of that CA-issued certificate would require minimal manual intervention.

To put it another way:

Let’s Encrypt

Let’s Encrypt is a CA that provides SSL certificates using the ACME protocol. They’re a non-profit organisation who describe themselves as ‘a free, automated and open certificate authority’. They’re also backed directly by the aforementioned ISRG. Their service enables automated generation, installation and renewal of SSL certificates with no downtime. Typically, this is done using a client called Certbot. Install Certbot on your web server, and it runs in the background and manages all aspects of your SSL certificates and the related configuration for your server.

For example, if you’re using nginx on CentOS, you’d run the following:

yum -y install yum-utils
yum-config-manager —enable rhui-REGION-rhel-server-extras rhui-REGION-rhel-server-optional
sudo yum install certbot-nginx
sudo certbot —nginx

Certbot will give you some instructions to follow to get your app and certificates set up correctly. You’d also need to add a recurring task using cron or systemd that executes certbot renew to enable auto-renewal.

That’s it. Run a few commands, and your app is secured with a free, auto-renewing SSL certificate. Adding the above commands to whatever automation scripts you’re using to build your environment (you have those, right?) makes enabling SSL for your application a trivial task.

While that’s a huge step forward, we can take it even further.


kube-lego

I’m not scared of containers or Kubernetes or cloud platforms, but I’m scared of insecure traffic

As part of moving our app from in-house infrastructure to the Google Cloud Platform, we can now leverage Kubernetes to remove all manual setup for SSL. When we add a new Ingress resource (essentially a load-balancer) to our GCP cluster, with no manual intervention we automatically generate and install an SSL certificate. We can do this across environments with no extra work, and the whole thing just sits there in the background — no effort, no ongoing management. How? kube-lego.

Kube-lego is a wrapper around Let’s Encrypt specifically designed to be used with Kubernetes Ingress resources. Setup for kube-lego is pretty trivial:

  1. Create a new Kubernetes namespace
  2. Create a new ConfigMap to specify an email address and the Let’s Encrypt environment to use (staging or production)
  3. Create a new Deployment to run the kube-lego container

At this point, you’ll have the kube-lego application up and running in your cluster. It will create a new Let’s Encrypt account for the email address specified in the ConfigMap, then look for Ingress resources.

For an Ingress resource to be picked up by kube-lego, you need to add the following annotation to its Deployment spec:

metadata:
annotations:
kubernetes.io/tls-acme: “true”

Kube-lego will then request a certificate for each entry in the spec.tls block of the Ingress resource.

spec:  
tls:
- secretName: mysql-tls
hosts:
- phpmyadmin.example.com
- mysql.example.com
- secretName: postgres-tls
hosts:
- postgres.example.com

For the above spec, certificates will be created for phpmyadmin.example.com and mysql.example.com, and stored in the Kubernetes secret mysql-tls; a certificate for postgres.example.com will be stored in the postgres-tls secret. The relevant Ingress resource will then use those certificates for all incoming requests, and you’ll get a nice green padlock in the browser to show off your new-found security, and at least an A rating on SSLLabs. Kube-lego also handles the renewal of your certificates when required.

Of course, we’ve added the kube-lego setup to the deployment scripts for our app, with the host name in the Ingress spec inserted based on the environment we’re deploying. With that done, we can add new environments and have SSL from the get-go, without even having to think about it.


Wrap Up

So there we have it — simple, straightforward, free SSL, with very little setup and no ongoing maintenance. Whether you’re using Kubernetes or not, I’d encourage you to look into Let’s Encrypt, certbot and kube-lego, and consider if there’s a chance you may be able to leave the old world of certificate management behind. Oh, and if you do end up using Let’s Encrypt, consider using some of the money you saved to shoot them a donation — you’d be in good company.