Let’s Encrypt and HAProxy withDocker

Let’s Encrypt is a service that allow one to obtain SSL certificates signed by a trusted CA for free. Those have are valid for at most 90 days and then, those need to be renewed. The client to do so is called certbot.

There are a lot of examples, repos, images that describes how to use the certbot client, but there aren’t much explanation around it.

I’ve found a good simple example by Tai Lee (thanks a lot!) on github of how it can be done. The code forked from the original repository can be found here

There was a couple of things I really liked about it:

• It was using dockercloud/haproxy docker image which I already know works very well
• It was using certbot within an alpine based docker image
• Everything was configurable using environment variables

And some I didn’t like about the approach:

• It was using sleep in order to wait for the load balancer to be up and running.
• I also had some trouble worth mentioning before I was able to successfully obtain a valid SSL certificate from the basic demo

What we want to achieve

TL;DR;

This is what we’ll end up adding to our docker-compose file to protect the web service

version: "2"
services:
  haproxy:
    image: m21lab/haproxy:1.6.2
    links:
      - letsencrypt
      - web ## THIS IS THE SERVICE HOSTED BEHIND THE NEW CERTIFICATE
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    volumes_from:
      - letsencrypt
letsencrypt:
    image: m21lab/letsencrypt:1.0
    environment:
      - DOMAINS=YOUR_DOMAIN
      - EMAIL=admins@YOUR_DOMAIN
      - LOAD_BALANCER_SERVICE_NAME=haproxy
      # THIS IS CRUCIAL WHEN TESTING to avoid reaching
      # the 5 certificates limit per domain per week. 
      # You'll end up waiting a week before being able 
      # to regenerate a valid cert if you don't backup
      # the once generated
      - OPTIONS=--staging

  web:
    environment:
      - FORCE_SSL=yes
      - VIRTUAL_HOST=http://*,https://*
    image: dockercloud/hello-world:latest

What it does

HAProxy service

• Binds ports 80 and 443 publicly
  it should be the only service binding ports directly to the host
• Based on dockercloud/haproxy docker image
  It performs reconfiguration of HAProxy when containers are started/stopped, which is very useful when scaling a service.

NOTE: make sure you add the volume binding to /var/run/docker.sock, otherwise the service won’t be able to probe for added/removed services’ containers

volumes:
      - /var/run/docker.sock:/var/run/docker.sock

• Has access to letsencrypt service volumes
  It defines a volumes_from configuration to be able to access the volumes exposed by the letsencrypt service which contains account information and generated SSL certificates
• Generates a default SSL certificate on HAProxy start
  This is very useful to make sure you still have access to your services behind SSL even when letsencrypt service is not ready or properly configured somehow.
• Watches for certificates generated by the letsencrypt services
  When new certificates are detected, those are installed in /certs (default HAProxy certificates folder) as letsencrypt*.pem, then the HAProxy service is restarted to use them.

Let’s encrypt service

• Requires the HAProxy service
  Waits for the load balancer service to be listening on port 80 before starting
• Requires environment variables for configuration
  - DOMAINS: a ‘;’ seperated list of domain(s)/subdomain(s) to get certificates for
  - EMAIL: a recovery email in case you lose your account information
  - LOAD_BALANCER_SERVICE_NAME: the name of the service running HAProxy in the docker-compose file. It’s used to wait for the service to be up.
• Contains Let’s Encrypt certbot client
  This client is used to request certificates for a single or multiple domain(s)/subdomain(s)
• Uses certbot webroot plugin
  Different plugins are available, but webroot was chosen because it allowed to easily host the challenge for domain validation locally. It uses an already existing web server and all you need to specify to the certbot client is the root of that server so the challenges are created there.
• A daily cron job is defined to ensure the certificate is renewed when it expires

NOTE: A challenge is a file to hosted on the server which must be retrieved using the requested domain’s root to validate the server is part of the domain you request and SSL certificate for.
e.g.: mydomain.com

• Configured to receive challenge traffic from HAProxy
  All requests to */.well-known/acme-challenge/* are redirected to the letsencrypt service
• Holds account information and generated certificates
  Those are located in the volume /etc/letsencrypt

The whole certificate generation flow

1. The letsencrypt service starts
2. letsencrypt service waits for haproxy services to be listening on port 80
3. Once the haproxy service is up, it generates a temporary SSL certificate, installs it in /certs (default HAProxy certificates folder) then restarts HAProxy in order to use this new certificate for SSL connections
4. A file watcher is installed on /etc/letsencrypt/live folder by the haproxy service to be able to restart HAProxy when new certificates are received.
   NOTE: this folder is available to the haproxy service because of the volumes_from definition in the service
5. letsencrypt service creates an http server to hold the challenge files
6. certbot command is executed which generates the challenge file locally in the webroot folder
7. Let’s Encrypt servers receive the request and try to request the challenge file using the domain(s)/subdomain(s) defined in DOMAINS environment variable one at the time
8. Once the validation is successful, the certificates are generated by Let’s Encrypt, sent to the letsencrypt service and copied in /etc/letsencrypt/live
9. The haproxy service restarts itself because of the file watcher

Problems

1. The initial repository was creating the http server after a 60 seconds delay instead of starting it right away. The side effect was that the HAProxy configuration was not routing the port 80 traffic to the letsencrypt service.
   Solution: started the http server first thing, this way HAProxy configuration was always routing to the letsencrypt service
2. The VERY IMPORTANT --staging parameter
   Make sure you set the environment variable OPTIONS: --staging on the letsencrypt service until you are 100% sure you are configured properly and you want to get a real certificate. Otherwise you’ll reache the 5 certificates limit per domain per week and you’ll end up waiting a week before being able to regenerate a valid certificate if you didn’t backup the ones already generated
3. Instead of using the certbot webroot plugin, it would have been nice to use the standalone plugin which basically creates an http server itself and listen for incoming connections. For some reason (possibly the same as the problem with the http server being created too late), the traffic was not redirected by HAProxy properly. This also prevented the https challenge (tls-sni-01) instead of the normal http (http-01) because it’s not yet supported by the webroot plugin
4. We were hosting the VM on azure and we were trying to get a certificate for <our-dns-label>.eastus.cloudapp.azure.com. This did not work because we don’t own azure.com and we received an error regarding the ratelimit.
   Solution: have IT to map a CName (Canonical Name) record to define an alias to this URL and register this alias instead.

Possible improvements

1. Get https challenge to work behind the haproxy service using the same setup