Kubernetes TLS bootstrapping

Bootstrapping TLS based communication in Kubernetes

This post is about bootstrapping TLS based communication in Kubernetes for the sake of kubelets and nodes. There is of course Kubernetes documentation that directly supports TLS bootstrapping here https://kubernetes.io/docs/admin/kubelet-tls-bootstrapping/, and while the Kubernetes documentation is a must read, it’s not always something you can wholly rely on to fulfill your requirements; you might find yourself scouring over GitHub issues and Stack Overflow posts in effort to correct inconsistencies in relation to the specifics of your cluster. That said, this particular post is not written with the idea to replace the Kubernetes documentation on the subject, but rather to build context and add answers to questions that might otherwise not exist.

Note: This post is written with the consideration around Kubernetes 1.9.x “cluster from scratch”. If you’re using a version < 1.9.x, you’ll need to research any specific requirements there might be in getting the bootstrapping to work.

Also, this post assumes that you have a certificate authority created using cfssl, and correlating flags configured in etcd, the api-server, and kube-controller-manager so that there is encrypted and authenticated traffic between them all.


The goal

The first thing I should mention is what the goal of TLS bootstrapping is. TLS bootstrapping is designed to ease the ability of securely joining nodes to masters, which could also include the ability to perform node based auto-scaling. If you don’t use TLS bootstrapping, you either have to employ unencrypted communication between masters and nodes (bad), or you have to update TLS CSRs to include hostnames and IP addresses, then regenerate and distribute certificates and keys to each master and node (kludgy); this of course is when you are adding new nodes to the cluster.

Figure 1.0 is a diagram of the TLS bootstrap flow.

Figure 1.0 TLS bootstrap workflow

In Figure 1.0 there is a decision about kubeconfig. This is to say that the kubelet will first look for the kubeconfig file and if it finds it, then likely the kubelet and node are already properly configured and will join/connect to the cluster upon boot. If kubeconfig doesn’t exist, the kubelet will reference bootstrap.kubeconfig, establish required comms, and dynamically build kubeconfig upon successful bootstrapping.

Figure 1.1 is a screenshot of CSRs that were requested by the kubelet-bootstrap user of each node, as well as the node itself. The kubelet-bootstrap user CSRs are automatically approved, and the node CSRs are waiting for cluster-admin approval.

Note: The kubectl get nodes command returns no nodes. This is because the system:node CSRs have not yet been approved.

Figure 1.1 kubectl get nodes + kubectl get csr

Now that an example of the goal has been established, let’s get into the important items that makeup the process toward achieving the goal.

Token authentication

At the core of the bootstrap process is token authentication. If you haven’t yet read or worked with token authentication, then please review this link to the Kubernetes documentation for it: https://kubernetes.io/docs/admin/authentication/#static-token-file

As mentioned in the Kubernetes documentation for TLS bootstrapping, tokens can be generated using /dev/urandom and then added to a token.csv file along with the correlating username, UID, and groups.

head -c 16 /dev/urandom | od -An -t x | tr -d ' '

Above is the token generation command, and figure 2.0 below is the contents of a typical token.csv file. Don’t worry, the tokens are not legit and are only here to serve as an example. As you can see in this example, there are two entries: admin, and kubelet-bootstrap. The admin line supports the ability for remote admin authentication using kubectl. The user kubelet-bootstrap line is there for bootstrapping kubelets.

Note: The kubelet-bootstrap line includes a group/role called system:node-bootstrapper. If you have a cluster that supports RBAC and you run kubectl get clusterroles you should see this role, and it exists to allow the ability to create, get, list, and watch certificate signing requests. The kubelet-bootstrap user must be a part of this group in order for TLS bootstrapping to succeed.

Figure 2.0 token.csv

kubelet

The kubelet requires special configuration for bootstrapping. The following flags need to be passed to the kubelet to allow for a bootstrap config, as well as a post bootstrap config, and for certificate rotation.

--kubeconfig=/var/lib/kubelet/kubeconfig \ 
--bootstrap-kubeconfig=/var/lib/kubelet/bootstrap.kubeconfig

Notice that even though there’s a requirement for bootstrap.kubeconfig, that there’s still a need for kubeconfig. This is because the bootstrap.kubeconfig provides a user and token (in token.csv) with minimal permission to get the connection established to the api-server and kube-controller-manager. Once the connection is made and the bootstrap CSR is generated and signed by the kube-controller-manager, the kubelet will dynamically create the kubeconfig including a client-certificate and client-key, and then generate another CSR for the node…which will need to be manually approved before the node can be used.

The following figure 3.0 is a bootstrap.kubeconfig file…again, not to worry, the certificate data, server address, and token are all examples.

Figure 3.0 bootstrap.kubeconfig

Notice that the certificate data for the certificate authority is embedded; this is a requirement and is satisfied when generating the bootstrap.kubeconfig . using the following command:

kubectl config set-cluster ${project} \
--certificate-authority=tls/ca.pem \
--embed-certs=true \
--server=https://${kubernetes_public_address}:6443

In the command above, the project var correlates to the cluster name in figure 2.1, and the kubernetes public address correlates to the cluster server; these exist as variables in this command because discovery commands in this example are issued prior to this command.

The following figure 3.1 is the dynamically created kubeconfig…again, not to worry, the certificate data, server address, and client certificate and key are all examples. The client-certificate and client-key are both auto-generated by the kubelet and are subsequently used for all future authentications.

Figure 3.1 kubeconfig

kube-apiserver

Amongst other things, the kube-apiserver hosts config for authentication and authorization of kubelets, and the goal in-mind here is to acknowledge kubelets and nodes so they can be bootstrapped into the cluster, approved, and finally provisioned by the scheduler.

Once a token.csv file is created, it’s time to add the following flags to the kube-apiserver config:

--authorization-mode=Node,ABAC,RBAC \ 
--token-auth-file=/var/lib/kubernetes/token.csv \

The Node authorization mode allows the kubelet to perform API based operations so that it can be bootstrapped into the cluster. When it comes to TLS bootstrapping, the kubelet will contact the API server, and if the username and token that’s used references the system:node-bootstrapper role in the token.csv file, then the API server will permit the kubelet the ability to perform create, get, list, and watch actions with certificate requests, as illustrated in Figure 4.0. The ABAC and RBAC authorization methods are available to support the transitioning from ABAC to RBAC…RBAC is required for TLS bootstrapping due to cluster role integrations.

Note: The order of node authorization modes is crucial. If you put RBAC before Node, you will find a lot of RBAC DENY entries in your kube-apiserver logs. Details around this are opened and closed here.

Figure 4.0 system:node-bootstrapper clusterrole

kube-controller-manager

As touched on previously, the kube-controller-manager is responsible for signing all CSRs and thus plays a critical role in the process. The following flags will need to be set in the kube-controller-manager config:

--cluster-signing-cert-file=/etc/path/to/kubernetes/ca/ca.pem \
--cluster-signing-key-file=/etc/path/to/kubernetes/ca/ca.key \

RBAC

As mentioned previously, RBAC plays a critical role in TLS bootstrapping Without RBAC, the kubelet-bootstrap user cannot join the necessary cluster roles that enable node bootstrapping. Below are three important kubectl commands that need to be executed so that the kubelet-bootstrap user can bootstrap nodes, and have CSRs auto approved and auto-renewed (if auto-renewal is configured in the master).

kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap
kubectl create clusterrolebinding node-client-auto-approve-csr \
--clusterrole=system:certificates.k8s.io:certificatesigningrequests:nodeclient \
--group=system:node-bootstrapper
kubectl create clusterrolebinding node-client-auto-renew-crt \
--clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeclient \
--group=system:nodes

Once the above is complete, and the kube-apiserver, kube-controller-manager, and kubelet is configured, the kubelet should startup and perform the bootstrap process with the kube-apiserver, then the authorization flow should immediately happen between the kubelet and the kube-controller-manager.

If you’re new to RBAC, make sure you create a cluster role binding to add your admin user (in the token.csv file) to your cluster-admin cluster role. If you don’t do this, you’ll struggle to establish several RBAC configurations down the road. I only mention this because it threw me off originally when I first started working with RBAC, and there’s not much out there to let you in on this detail.

kubectl create clusterrolebinding cluster-admin-users \
--clusterrole=cluster-admin \
--user=admin

Result

Figure 5.0 comes back to the output of kubectl get csr as illustrated in Figure 1.1, but it also shows what happens when the node certificates are manually approved and thus are returned when kubectl get nodes is returned.

Figure 5.0 kubectl CSR approval for nodes

Auto Scaling

You might still be wondering to yourself how to get full auto scaling for nodes and kubelets to happen, and this is a possibility by adding the system:authenticated group to your cluster-admin based cluster role binding. However, this would seem unadvisable and an anti-pattern with Kubernetes security and scaling. The pattern for scaling with Kubernetes is to be a secured and controlled one, and the point of TLS bootstrapping is to avoid the requirement of having to update CSRs, regenerate certificates, and then redistribute. The manual step of approving the node CSR adds security and control to your cluster, but that said, there could be a cluster role that will provide secure auto-approval and rotation of node CSRs.

TL;DR

  1. Read the Kubernetes documentation on the subject
  2. Generate a token for kube-bootstrapper and add it to your token.csv
  3. Create a bootstrap.kubeconfig
  4. Configure the kubelet to reference both kubeconfig and bootstrap.kubeconfig
  5. Configure the kube-apiserver authorization modes and token auth file
  6. Configure the kube-controller-manager with the certificate and key signing files
  7. Configure the RBAC cluster role bindings