Scaling SSL certificates with Kubernetes and Let’s Encrypt

How we built an automated pipeline for issuing SSL certificates using Kubernetes, AWS, and Let’s Encrypt

At Qwilr, we give our customers the ability to create their sales and marketing documents as web pages. They look great, impress clients, and feel like they’re simply an extension of your website, namely through having content served via their own custom domain. As an example, “Fancy Corp” can serve their Qwilr pages at “proposals.fancycorp.com”.

Until recently, we weren’t able to support secure pages for our customers using this strategy because we didn’t have a good solution for automatic SSL certificate generation. And this wasn’t surprising — I’ve been dabbling in SSL setup since 2001 and it’s continued to be a complex task to iron out.

We were looking for a solution that could create an SSL certificate for each of our enabled custom domains automatically, and one that was equally independent from our system and require little maintenance.

Enter Let’s Encrypt (largely inspired by Readme.io’s post) and OpenResty’s fantastic lua-ssl-nginx-module.

Let’s Encrypt

If you haven’t heard of Let’s Encrypt, it’s a certificate authority with an API that provides short lifetime certificates on demand. I can request a certificate for my domain, and download it via an API immediately, provided I can prove I own that domain using the Automatic Certificate Management Environment (ACME) protocol. There is a worthwhile explanation here.

Setup
From a high level, our NGINX SSL reverse proxy setup stands like this:

All the proxy side is run in AWS inside our Kubernetes cluster running on EKS. This was my first experience with setting up a service for production using Kubernetes directly and it was quite a steep and fun learning curve.

Inside the custom domain Kubernetes environment we run:

• Load balancer service: internet facing endpoint that distributes traffic to nginx deployment
• Nginx Lua Deployment: does handshake with Let’s encrypt and serves content for our custom domains
• Redis Service: store for certificates created from Let’s Encrypt
• StatefulSet backing the redis instance: durable storage for redis

NGINX Lua SSL Setup

Open Resty is a web platform that integrates Nginx core, LuaJIT, and many Lua based libraries. This is a great answer to many micro-service and routing requirements and fits our purpose perfectly. We even have the SSL module which talks to Let’s Encrypt. Here’s our nginx config.

The only real new code to the Qwilr system was an API endpoint which returns either 200 or 404 depending on whether the requested domain is respectively, one of ours or not.

Kubernetes

The nuts and bolts of the system are captured in a single Kubernetes configuration here:

A notable call-out on this setup— I use this docker image to build on top of: openresty/openresty:latest-xenial and by using these config sets and different namespaces, we can easily create a completely self-contained dev version of this environment.

Learnings and outcomes

This project highlighted the value of Let’s Encrypt in enabling the web to be more secure. My first job out of university was working for RSA Security on the SSL-C product, and since 2001, I’ve been amazed at the difficulty in getting setup to do SSL. ACME and Let’s Encrypt provide an excellent solution to a long-standing problem. We also decided to give $1 per certificate served from Let’s Encrypt in support of this excellent service.

Perhaps the biggest learning was about Kubernetes. For years now I’ve heard it’s a fantastic way to run infrastructure, but doing this project allowed me to create something that could run parallel to a production deployment without having to do a large “lift-and-shift”, but still remain core to our business. It also allows for a fault-tolerant and scalable “auto-ssl” system to enable custom domains for Qwilr enterprise customers.

And on the flip side, the results for our customers are great. They avoid Chrome’s very frightening “not secure” warning in the address bar (because no one wants that!) and get secure content served from a domain of their choice.