Deploy Certificate Authority Service on Kubernetes

This post is a short guide on how to deploy cfssl as Certificate Authority (CA) service on Kubernetes.

Now you might ask why would anyone want to do that given that Kubernetes comes with a CA out of the box. This is a good point and the Kubernetes implementaion may be good enough for some use cases. However, note that in order to use it you first need to configure your cluster properly and make sure that the key for signing certificates is on all of your master nodes, what can be non trivial.

Another issue with the Kubernetes CA is that you either need to manually approve certificate signing requests or write a certificate controller that will do it automatically.

The point I’m trying to make is that the setup and usage of the Kubernetes CA is non-trivial. You might need to dig through the Kuberentes documentation, which I personally find hard to follow or incomplete sometimes (DISCLAIMER: I’m not very experienced in setting up and running Kubernetes, so please take this with a grain of salt). Even when succeeding in setting it up it may still not meet your exact needs. Whereas the setup of cfssl CA is easy and flexible as you will see soon.

Cfssl is a an open source tool built by Cloudflare for generating, signing and bundling certificates and much more. It is described as “TLS swiss army knife” in the README of its repository. Among other things it can run as an http server that acts as a CA, that is it signs certificate signing requests that you send to it.

Given that cfssl is an open source project it also has an offical Docker image published on the Docker hub. This is great as we can just use it in our Kubernetes manifest. Now the only thing we need to figure out is how to run it. We can start the http server with the following command.

cfssl serve -address=0.0.0.0 -port=8080 -config=config.json         -ca=ca.pem -ca-key=ca-key.pem

This will start the server on the port 8080. You can make the following POST request to retrieve the certificate containing the public key of the CA. This will be the content of the ca.pem file.

curl -d '{}' -H "Content-Type: application/json" -X POST localhost:8080/api/v1/cfssl/info

To get a certificate signed by the CA you can run the following command.

cfssl gencert -config=config-client.json -profile=client-server -remote=localhost:8080 csr.json | cfssljson -bare node

The contents of the certificate signing request file (csr.json) should look something like the following example taken from the wiki of the cfssl repository.

{
  "hosts": [
    "example.com",
    "www.example.com"
  ],
  "CN": "www.example.com",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [{
    "C": "US",
    "L": "San Francisco",
    "O": "Example Company, LLC",
    "OU": "Operations",
    "ST": "California"
  }]
}

There we specify the Common Name (CN) for the certificate, the hosts it will be valid for, the details of the generated key like the signing algorithm to use and its size in bytes and the names that will appear on the certificate. The contents of the config-client.json file should be as in the example shown below. In this file we specify what authentication key and signing profile to use. I will explain these later when I go through the config file for the CA.

{
  "signing": {
    "profiles": {
      "client-server": {
        "auth_remote": {
          "auth_key": "client",
          "remote": "ca"
        }
      }
    }
  },
  "auth_keys": {
    "client": {
      "type": "standard",
      "key": "D08E2AD3153827496ADDA6FB104624B2"
    }
  },
  "remotes": {
    "ca": "0.0.0.0:8080"
  }
}

Now that we know how to use the service it is a good time to describe all the parameters we use to run the http server with. Address and port are self explanatory. The -ca and -ca-key flags point to a pair of certificate and key files, that will be used for signing by the CA. There are various ways to generate these, but as cfssl is a powerful tool we can use it to do that by running the following command.

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

Here the ca-csr.json is a file with the following contents.

{
  "CN": "My Personal CA",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "ca": {
    "expiry": "17520h"
  }
}

There again CN is the Common Name of the certificate authority and key specifies the details of the key i.e. what signing algorithm to use and what should be the key size in bytes. In the ca field we specify the expiry of the certificate. Running the command will produce two files ca.pem and ca-key.pem which we can use directly with our CA. Note that the generated certificate is self signed. This means that it is signed by the CAs own key, so browsers will complain about certificates signed by this CA as they don’t know it.

Finally, it’s time to talk about the config file. It is a JSON file that should look something like the example below. The config has two parts signing and auth_keys. The later contains a list of authentication keys that clients can use to access the service. Note that the key should be base64 encoded 32 byte string. The signing section contains signing profiles for generating different kinds of certificates. In our particular use case we wanted to force users to specify the profile they are using, so we generate a random key every time the service starts and require this key in order to use the default profile, what makes it pretty much inaccessible.

There are two signing profiles specified in the example config, one for server usage and another one for client usage. Both profiles use the same authentication key. Furthermore we specify the expiry for the certificates signed using these profiles and what they can be used for. You can find full explanation of the usages here.

{
  "signing": {
    "default": {
      "auth_key": "unused",
      "expiry": "1h"
    },
    "profiles": {
      "client": {
          "expiry": "%%%EXPIRY_CLIENT_HOURS%%%h",
          "usages": [
            "critical",
            "digital signature",
            "key encipherment",
            "client auth",
            "signing"
          ],
          "auth_key": "client"
        },
        "server": {
          "expiry": "%%%EXPIRY_SERVER_HOURS%%%h",
          "usages": [
            "critical",
            "digital signature",
            "key encipherment",
            "server auth",
            "client auth",
            "signing"
          ],
          "auth_key": "client"
        }
    }
  },
  "auth_keys": {
    "client": {
      "type": "standard",
      "key": "%%%AUTH_KEY%%%"
    },
    "unused": {
      "type": "standard",
      "key": "%%%RANDOM_KEY%%%"
    }
  }
}

Now you probably noticed that some values are not specified in the config file. We populate these from environment variables in an init container using sed, what is a small utility that you can find in most linux distributions and also in the busybox docker image. The complete script is shown below.

#!/bin/bash

RANDOM_KEY=$(hexdump -n 16 -e "4/4 \"%08X\" 1 \"\n\"" /dev/random);

# This assumes that this config map is mounted in /scripts folder.
cat /scripts/config-template.json |
sed "s/%%%AUTH_KEY%%%/${AUTH_KEY}/g" |
sed "s/%%%RANDOM_KEY%%%/${RANDOM_KEY}/g" |
sed "s/%%%EXPIRY_CLIENT_HOURS%%%/${EXPIRY_CLIENT_HOURS}/g" |
sed "s/%%%EXPIRY_SERVER_HOURS%%%/${EXPIRY_SERVER_HOURS}/g" > /config/config.json;

If you are not familiar with init containers, then you have nothing to worry about, as they are just normal containers that run before the main app containers. They are usually used to set up configuration, fetch resources or wait for other services to start up. They can contain scripts and resources that you don’t want to or can’t have in your main container. You can have any number of init containers, but they always have to run to completion, so that once they finish your app can start. You can read more about them here.

The main reason why we are computing the config in an init container is that we want to store the AUTH_KEY in a Kubernetes secret to keep it secure.

This pretty much covers most of the setup, so now it’s time to put it all together. You can find the complete manifests here. We store the template for the config and the bash script that populates it in a config map. Notice the use of the "|" character in the manifest of the config map. This is yaml syntax for interpreting the following lines (except for the indentation) as a string while preserving line breaks.

The config map is mounted as a /scripts folder in the init container that uses the busybox docker image. This container runs the initialisation script and stores the computed config file in a pod local volume that gets destroyed together with the pod. This volume is initialised as an empty folder. The AUTH_KEY and server and client expiry variables are populated from secrets and config map keys respectively.

The main container uses the cfssl docker image and runs the command specified earlier. One last thing to note is that it mounts a secret containing the certificate and the key for the CA as a /certs folder.

The service is setup to run as a deployment as it doesn’t have any state that it needs to preserve. We run two pods for high availability, but you can probably get away with just one. If on the other hand you want to make sure that you have at least one pod running at all times, you can specify a pod disruption budget that will only allow one pod to be down at a time.

To create the secrets used in the manifests you can run the following commands.

kubectl create secret generic ca-certs --from-file=ca.pem --from-file=ca-key.pem
kubectl create secret generic ca-auth-key --from-literal=auth.key=$(hexdump -n 16 -e '4/4 "%08X" 1 "\n"' /dev/random)

And that’s it. You are now ready to run your own Certificate Authority service. Thank you for making it all the way. If you liked this post then please hit the clap button, if you have any questions then please leave them in the comments below.