Alternative ACME via cert-manager

Disclaimer; I love LetsEncrypt. Like, I really love it. It’s opened up SSL to the world and we’re better off as a result. But sometimes, their rate limits suck.

Context: I want to use ACME certificates for a CI environment for end-to-end tests. The domains will be `app.<branch-name>.<app>.<domain>` and `admin. <branch-name>.<app>.<domain>`. So wildcard certificates are not an option here as it’d be a multi-level wildcard (yuck!). There are alternative ways to do it, but I want to keep my CI environment the same as production & staging!

After a bit of research, I stumble upon ZeroSSL — a free ACME provider that seems to go a bit under the radar. It comes with all the things I needed. No rate limits, unlimited certificates and something extra cool; 1 YEAR CERTIFICATES! Wow, nice.

So in this tutorial, I’m going to walk you through configuring cert-manager on Kubernetes to work with ZeroSSL so you can generate and deploy TLS Certificates for your ingress(s).

I’m going to assume you’ve installed cert-manager onto your cluster already. In this tutorial, I’ll walk you through how to create the cluster-issuer to use with ZeroSSL, and the credentials from ZeroSSL to authenticate between your cluster and their acme API. I use DNS to verify my domains so any references to Route53 is for that.

First things first, let’s get a EAB from ZeroSSL (ps you’ll need the $10 a month subscription to use some of this).

Head over to https://app.zerossl.com/developer

Generate a EAB and note these down, you’ll only see them once!

Next! Let’s do some kubernetes magic…

Your skeleton YAML file (ps change namespace in the secret from kube-system to the namespace in which you’re running cert-manager if necessary):

apiVersion: v1
kind: Secret
metadata:
  namespace: kube-system # The NS you're running cert-manager in
  name: zero-sll-eabsecret
stringData:
  secret: YOUR_HMAC_HERE
---
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: ssl-prod
spec:
  acme:
    # The ACME server URL
    server: https://acme.zerossl.com/v2/DV90
    externalAccountBinding:
      keyID: YOUR_EAB_KID
      keySecretRef:
        name: zero-sll-eabsecret
        key: secret
      keyAlgorithm: HS256
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: zerossl-prod
       # Enable the HTTP-01 challenge provider
    solvers:
    - dns01:
        route53:
          region: eu-west-2
          accessKeyID: ${AWS_ACCESS_KEY_ID}
          secretAccessKeySecretRef:
            name: cert-manager-route53-aws
            key: AWS_SECRET_KEY_ID

Walking through it bit by bit.

1. EAB secret. So unlike some other ACME apps that ZeroSSL has implemented with, cert-manager isn’t one of them. So we’ll be using EAB as a way to authenticate and communicate with ZeroSSL from our cluster. In essence it’s like a key and secret to use with the ACME API.

This secret is to store the EAB HMAC you’ll get from ZeroSSL. An alternative route to creating your EAB Secret is this kubectl command:

kubectl -n kube-system create secret generic zero-sll-eabsecret — from-literal secret="<YOUR HMAC HERE>"

If you’re not running cert-manager in kube-system, update any references of kube-system to the namespace you’re using.

Alternatively, this is how you create the secret using YAML:

apiVersion: v1
kind: Secret
metadata:
  namespace: kube-system
  name: zero-sll-eabsecret
stringData:
  secret: YOUR_HMAC_HERE

2. The main magic

So we’re creating here a ClusterIssuer (so it’s cluster-wide and can be used in any namespace we so wish).

server: https://acme.zerossl.com/v2/DV90

We specify the server to use (source: https://zerossl.com/documentation/acme/)

externalAccountBinding:
      keyID: YOUR_EAB_KID
      keySecretRef:
        name: zero-sll-eabsecret
        key: secret
      keyAlgorithm: HS256

YOUR_EAB_KID should be replaced with your EAB KID provided to you above by ZeroSSL (hint: FYI shouldn’t be base64 encoded).

At the bottom I’ve also specified to use DNS solvers to verify my domain, there’s plenty of alternatives you can read about those here.

    solvers:
    - dns01:
        route53:
          region: eu-west-2
          accessKeyID: ${AWS_ACCESS_KEY_ID}
          secretAccessKeySecretRef:
            name: cert-manager-route53-aws
            key: AWS_SECRET_KEY_ID

Apply your files

kubectl apply -f filename.yaml

You can then specify in your ingress’s an annotation like so, and cert-manager will jump in and get a certificate generated for you:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: myingress
  annotations:
    cert-manager.io/cluster-issuer: ssl-prod

Having issues?

You can troubleshoot by doing some of the following:

1 — Watch your certificate status

kubectl get certificates -A

2 — Make sure that the acme has registered ok

kubectl get ClusterIssuers -o yaml

this should give you a section like so:

status:
    acme:
      uri: https://acme.zerossl.com/v2/DV90/account/redacted
    conditions:
    - lastTransitionTime: "2020-08-17T12:19:23Z"
      message: The ACME account was registered with the ACME server
      reason: ACMEAccountRegistered
      status: "True"
      type: Ready

3 — Look at the cert-manager logs! Find the name of the cert-manager pod and do

kubectl -n <namespace> logs -f <pod-name>

Have fun!