Kubernetes TLS bootstrapping

Bootstrapping TLS based communication in Kubernetes

Todd Rosner
Feb 19, 2018 · 8 min read

This post is about bootstrapping TLS based communication in Kubernetes for the sake of kubelets and nodes. There is of course Kubernetes documentation that directly supports TLS bootstrapping here https://kubernetes.io/docs/admin/kubelet-tls-bootstrapping/, and while the Kubernetes documentation is a must read, it’s not always something you can wholly rely on to fulfill your requirements; you might find yourself scouring over GitHub issues and Stack Overflow posts in effort to correct inconsistencies in relation to the specifics of your cluster. That said, this particular post is not written with the idea to replace the Kubernetes documentation on the subject, but rather to build context and add answers to questions that might otherwise not exist.

Note: This post is written with the consideration around Kubernetes 1.9.x “cluster from scratch”. If you’re using a version < 1.9.x, you’ll need to research any specific requirements there might be in getting the bootstrapping to work.

Also, this post assumes that you have a certificate authority created using cfssl, and correlating flags configured in etcd, the api-server, and kube-controller-manager so that there is encrypted and authenticated traffic between them all.


The goal

Figure 1.0 is a diagram of the TLS bootstrap flow.

Image for post
Image for post
Figure 1.0 TLS bootstrap workflow

In Figure 1.0 there is a decision about kubeconfig. This is to say that the kubelet will first look for the kubeconfig file and if it finds it, then likely the kubelet and node are already properly configured and will join/connect to the cluster upon boot. If kubeconfig doesn’t exist, the kubelet will reference bootstrap.kubeconfig, establish required comms, and dynamically build kubeconfig upon successful bootstrapping.

Figure 1.1 is a screenshot of CSRs that were requested by the kubelet-bootstrap user of each node, as well as the node itself. The kubelet-bootstrap user CSRs are automatically approved, and the node CSRs are waiting for cluster-admin approval.

Note: The kubectl get nodes command returns no nodes. This is because the system:node CSRs have not yet been approved.

Image for post
Image for post
Figure 1.1 kubectl get nodes + kubectl get csr

Now that an example of the goal has been established, let’s get into the important items that makeup the process toward achieving the goal.

Token authentication

As mentioned in the Kubernetes documentation for TLS bootstrapping, tokens can be generated using /dev/urandom and then added to a token.csv file along with the correlating username, UID, and groups.

head -c 16 /dev/urandom | od -An -t x | tr -d ' '

Above is the token generation command, and figure 2.0 below is the contents of a typical token.csv file. Don’t worry, the tokens are not legit and are only here to serve as an example. As you can see in this example, there are two entries: admin, and kubelet-bootstrap. The admin line supports the ability for remote admin authentication using kubectl. The user kubelet-bootstrap line is there for bootstrapping kubelets.

Note: The kubelet-bootstrap line includes a group/role called system:node-bootstrapper. If you have a cluster that supports RBAC and you run kubectl get clusterroles you should see this role, and it exists to allow the ability to create, get, list, and watch certificate signing requests. The kubelet-bootstrap user must be a part of this group in order for TLS bootstrapping to succeed.

Image for post
Image for post
Figure 2.0 token.csv

kubelet

--kubeconfig=/var/lib/kubelet/kubeconfig \ 
--bootstrap-kubeconfig=/var/lib/kubelet/bootstrap.kubeconfig

Notice that even though there’s a requirement for bootstrap.kubeconfig, that there’s still a need for kubeconfig. This is because the bootstrap.kubeconfig provides a user and token (in token.csv) with minimal permission to get the connection established to the api-server and kube-controller-manager. Once the connection is made and the bootstrap CSR is generated and signed by the kube-controller-manager, the kubelet will dynamically create the kubeconfig including a client-certificate and client-key, and then generate another CSR for the node…which will need to be manually approved before the node can be used.

The following figure 3.0 is a bootstrap.kubeconfig file…again, not to worry, the certificate data, server address, and token are all examples.

Image for post
Image for post
Figure 3.0 bootstrap.kubeconfig

Notice that the certificate data for the certificate authority is embedded; this is a requirement and is satisfied when generating the bootstrap.kubeconfig . using the following command:

kubectl config set-cluster ${project} \
--certificate-authority=tls/ca.pem \
--embed-certs=true \
--server=https://${kubernetes_public_address}:6443

In the command above, the project var correlates to the cluster name in figure 2.1, and the kubernetes public address correlates to the cluster server; these exist as variables in this command because discovery commands in this example are issued prior to this command.

The following figure 3.1 is the dynamically created kubeconfig…again, not to worry, the certificate data, server address, and client certificate and key are all examples. The client-certificate and client-key are both auto-generated by the kubelet and are subsequently used for all future authentications.

Image for post
Image for post
Figure 3.1 kubeconfig

kube-apiserver

Once a token.csv file is created, it’s time to add the following flags to the kube-apiserver config:

--authorization-mode=Node,ABAC,RBAC \ 
--token-auth-file=/var/lib/kubernetes/token.csv \

The Node authorization mode allows the kubelet to perform API based operations so that it can be bootstrapped into the cluster. When it comes to TLS bootstrapping, the kubelet will contact the API server, and if the username and token that’s used references the system:node-bootstrapper role in the token.csv file, then the API server will permit the kubelet the ability to perform create, get, list, and watch actions with certificate requests, as illustrated in Figure 4.0. The ABAC and RBAC authorization methods are available to support the transitioning from ABAC to RBAC…RBAC is required for TLS bootstrapping due to cluster role integrations.

Note: The order of node authorization modes is crucial. If you put RBAC before Node, you will find a lot of RBAC DENY entries in your kube-apiserver logs. Details around this are opened and closed here.

Image for post
Image for post
Figure 4.0 system:node-bootstrapper clusterrole

kube-controller-manager

--cluster-signing-cert-file=/etc/path/to/kubernetes/ca/ca.pem \
--cluster-signing-key-file=/etc/path/to/kubernetes/ca/ca.key \

RBAC

kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap
kubectl create clusterrolebinding node-client-auto-approve-csr \
--clusterrole=system:certificates.k8s.io:certificatesigningrequests:nodeclient \
--group=system:node-bootstrapper
kubectl create clusterrolebinding node-client-auto-renew-crt \
--clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeclient \
--group=system:nodes

Once the above is complete, and the kube-apiserver, kube-controller-manager, and kubelet is configured, the kubelet should startup and perform the bootstrap process with the kube-apiserver, then the authorization flow should immediately happen between the kubelet and the kube-controller-manager.

If you’re new to RBAC, make sure you create a cluster role binding to add your admin user (in the token.csv file) to your cluster-admin cluster role. If you don’t do this, you’ll struggle to establish several RBAC configurations down the road. I only mention this because it threw me off originally when I first started working with RBAC, and there’s not much out there to let you in on this detail.

kubectl create clusterrolebinding cluster-admin-users \
--clusterrole=cluster-admin \
--user=admin

Result

Image for post
Image for post
Figure 5.0 kubectl CSR approval for nodes

Auto Scaling

TL;DR

  1. Generate a token for kube-bootstrapper and add it to your token.csv
  2. Create a bootstrap.kubeconfig
  3. Configure the kubelet to reference both kubeconfig and bootstrap.kubeconfig
  4. Configure the kube-apiserver authorization modes and token auth file
  5. Configure the kube-controller-manager with the certificate and key signing files
  6. Configure the RBAC cluster role bindings

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store