Installing cert-manager on a gcloud K8s cluster

A week or two back I had the task of setting up SSL on all our environments at work, it wasn’t as straight forward as I expected so I’ve written up some notes in hopes that it helps others or future me.

We run everything on GKE, with a gateway graphql server in front of our backend micro services. So we only need a single valid certificate per cluster. Google manages SSL and custom domains for App engine but not their Kubernetes offering, so we have to validate the certificate our self.

I wanted to use letsencrypt, because it’s free and there is a bunch of tooling around setting it up. Kuberenete’s is no different, there are at least 5 different projects managing certificates with Kubernetes. I was lead to cert-manager, as it is the predessesor of the popular kube-lego project. Cert manager is simply a service in your cluster that reads your certificate (stored as a secret) and prior to the certificate expiring (letsencrypt secrets last 90 days) it makes a API call to letsencrypt for a new certificate and then stores the secret for you like nothing happened. 👏

Installing cert-manager:

All of our services are either in house or google cloud services so there hadn’t been a need to use helm. Helm is a package manager for kubernetes and the primary means of installing cert-manager.

We didn’t want to install Helm in our cluster just to manage a single service so we I decided to install it manually. There aren’t currently docs on this but you can do so by running:

git clone https://github.com/jetstack/cert-manager \
&& cd cert-manager \
kubectl apply -f docs/deploy/without-rbac

You should now have cert-manager up and running.

Now there are several ways for cert-manager to provision new certificates. These are configured by deploying Issuer object types.

The issuers can respond to HTTP challenges where lets encrypt expects to receive a response when it hits a certain URL. However this involves setting routing via kubernetes ingress, which seemed like a pain. (there is a great guide on using that method here: https://github.com/ahmetb/gke-letsencrypt/)

I instead went down the route of DNS validation. With this method letsencrypt sets a challenge, for a specific DNS record to be available at your domains address. We are managing our DNS records with google DNS and luckily cert-manager has google DNS builtin.

First you’ll want to verify the ownership of your domain with google. Unfortunately the domains supplied by google GKE are no good as you don’t control the DNS and there for can’t do DNS validation. For clarity these are the <GOOGLE__PROJECT_ID>.cloud.goog domains.

Once you have verified your domain, set up your k8s cluster on a static IP and add a DNS A record to point to the IP. Make sure you can reach your cluster with the new domain.

Now we are ready to configure cert-manager to validate certificates via letsencrypt and google DNS.

Your issuer.yml will look something like this:

# issuer.yaml
apiVersion: certmanager.k8s.io/v1alpha1
kind: Issuer
metadata:
  name: letsencrypt-prod
  namespace: default
spec:
  acme:
    # The ACME server URL
    server: https://acme-staging.api.letsencrypt.org/directory
    # server: https://acme-v01.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: {{YOUR_EMAIL}}
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod
    # ACME DNS-01 provider configurations
    dns01:
      # Here we define a list of DNS-01 providers that can solve DNS challenges
      providers:
      - name: prod-dns
        clouddns:
          # A secretKeyRef to a google cloud json service account
          serviceAccountSecretRef:
            name: cert-manager-credentials 
            key: cert_manager_service_account.json
          # The project in which to update the DNS zone
          project: {{GOOGLE_PROJECT_ID}}

A couple things to notice, you need a kube secret cert-manager-credentials, this should contain a cert_manager_service_account.json which has access to read/write kuberenetes secrets and interact with the google DNS API. You will also need to update GOOGLE_PROJECT_ID & YOUR_EMAIL.

Next we want to create a Certificate, this tells cert-manager which issuer to use and what domains to validate.

# certificate.yaml
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: {{HOST}} 
  namespace: default
spec:
  secretname: tls-cert
  issuerRef:
    name: letsencrypt-prod
  commonName: {{HOST}}
  dnsNames:
  - {{HOST}} 
  - www.{{HOST}}
  acme:
    config:
    - dns01:
        provider: prod-dns
      domains:
      - {{HOST}} 
      - www.{{HOST}}

Replace {{ HOST }} with the domain in question.

Then deploy ‘em both:

kubectl create -f issuer.yaml certificate.yaml

Wait for cert-manager to do it’s magic and you should have a kubernetes secret called tls-cert with some fresh certs.

In our case we have a nginx proxy sitting in front of everything so we just mounted the certs as a volume

# deployment.yaml
# [START volumes]
 - name: nginx-ssl
  secret:
    items:
      - key: tls.crt
        path: nginx.crt
      - key: tls.key
        path: nginx.key
        secretName: tls-cert
# [END volumes]

And then mount them into the ssl/ directory in our nginx container.

# deployment.yaml
volumeMounts:
  - name: esp-instance-credentials
    mountPath: /secrets/esp
    readOnly: true
  - name: nginx-ssl
    mountPath: /etc/nginx/ssl
    readOnly: true

Now go to https://{{HOST}} and you should see a ...invalid certificate error. This is because we validated our certificates with letencrypts staging API. To rectify this change the server: in issuer.yam https://acme-v01.api.letsencrypt.org/directory.

Then redeploy your issuer:

kubectl create -f issuer.yaml

And delete the previous certificate, so that cert-manager will know to create a new one.

kubectl delete secret tls-cert

Enjoy your certificates 🍵