Learn How to Use TLS Bootstrapping in Kubernetes

Setting up a master and worker node for TLS bootstrapping can be tricky because many configuration steps are involved, so we decided to walk you through the process of enabling this useful feature in Kubernetes. Let’s get started!

What Is TLS Bootstrapping?

Kubernetes recommends making communication between node components such as kubelet and kube-proxy and master components private and secure. To achieve this, we need to distribute TLS certificates for node components and sign them with the master CA. Prior to Kubernetes v1.4, users had to do most of the TLS bootstrapping work such as issuing and distributing certificates manually. In particular, the following steps were involved:

1. Creating a Certificate Authority (CA) key and a certificate
2. Distributing the CA certificate to the master(s)
3. Creating a certificate and a key for each kubelet.
4. Signing the kubelet certificate using the CA key on the master
5. Distributing the kubelet key and the signed certificate to the node where the kubelet will be running

A certificate request and signing API was introduced in Kubernetes v1.4 to simplify this process. More specifically, it automates steps 3 forward. Thus, TLS bootstrapping may be described as a mechanism for the automatic retrieval of certificates from the API to allow newly created nodes(s) to communicate with master(s) securely. This feature became Generally Available in Kubernetes v. 1.12.

Tutorial

For the purpose of this article, we will use kubeadm to provision a master node, create a worker node, set up it manually, and connect to the master. Instructions on creating a single-master K8s cluster can be found here.

Assuming that a node has been created, we need the following information:

1. master host:port
2. a bootstrap token
3. a CA certificate

First two can be obtained from the kubeadm init output down below:

Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a Pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join <master-ip>:<master-port> --token <bootstrap-token> --discovery-token-ca-cert-hash sha256:<ca-cert-hash>

A CA cert can be found on a master node in /etc/kubernetes/pki/ca.crt , which is its default location.

To bootstrap a new node, we need only two binaries: kubectl and kubelet .
Download those binaries from here and unpack them:

wget https://dl.k8s.io/v1.13.4/kubernetes-server-linux-amd64.tar.gz tar -xvf kubernetes-server-linux-amd64.tar.gz 
sudo cp kubernetes/server/bin/kubelet /usr/bin 
sudo cp kubernetes/server/bin/kubectl /usr/bin

A container runtime is also a required node component, so don’t forget to install Docker or any other container runtime you prefer (e.g., rkt):

sudo apt-get update sudo apt install -y docker.io

Next, create a folder for storing Kubernetes-related files:

sudo mkdir /etc/kubernetes
sudo mkdir /etc/kubernetes/manifests
sudo mkdir /etc/kubernetes/pki

Copy the ca.crt from the master to /etc/kubernetes/pki

Now, we are almost prepared for the TLS bootstrapping of a new node. We do need two additional things, though:

1. A path to kubeconfig that does not exist yet and will be stored there
   after successful bootstrap:--kubeconfig
2. A path to bootstrap config: --bootstrap-kubeconfig

We can either create a bootstrap config manually using kubectl config command or just copy the kubeconfig from a master node:

sudo kubectl config --kubeconfig=/etc/kubernetes/bootstrap-kubeconfig set-cluster kubernetes --server='https://<master-host>:<master-port>' --certificate-authority=/etc/kubernetes/pki/ca.crt --embed-certs=true 
sudo kubectl config --kubeconfig=/etc/kubernetes/bootstrap-kubeconfig set-credentials tls-bootstrap-token-user --token=<bootstrap-token> 
sudo kubectl config --kubeconfig=/etc/kubernetes/bootstrap-kubeconfig set-context tls-bootstrap-token-user@kubernetes --user=tls-bootstrap-token-user --cluster=kubernetes 
sudo kubectl config --kubeconfig=/etc/kubernetes/bootstrap-kubeconfig use-context tls-bootstrap-token-user@kubernetes

The last step we need to do is to create a systemd
file in /etc/systemd/system/kubelet.service to run kubelet as a daemon.

We use the RotateKubeletClientCertificate feature gate that enables certificate rotation. This Beta feature enables kubelet to update a certificate each time it’s going to expire.

[Service] 
Restart=always 
ExecStart=/usr/bin/kubelet --kubeconfig=/etc/kubernetes/kubeconfig \\ 
--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubeconfig \\ 
--pod-manifest-path=/etc/kubernetes/manifests \\ 
--feature-gates=RotateKubeletClientCertificate=true \\ 
--rotate-certificates 
[Install] 
WantedBy=multi-user.target

Reload systemd daemon and enable kubelet service to run on startup.

systemctl daemon-reload
systemctl enable kubelet
systemctl start kubelet

Finally, check that node is up and running in Ready state:

kubectl get  no
NAME         STATUS   ROLES    AGE   VERSION
instance-1   Ready    master   15m   v1.13.4
instance-2   Ready    <none>   30s   v1.13.4

Conclusion

That’s it! We have successfully enabled TLS bootstrapping of a new node. This automatic feature makes connecting nodes to the cluster and securing their communication with the kube-apiserver much easier. In particular, we do not need to manually create kubelet certificates and sign them with the master certificates because this process is automatically handled by the Kubernetes TLS API.

Below is the list of references that will help you install a single-master K8s cluster with kubeadm and learn more about TLS bootstrapping:

• https://kubernetes.io/docs/setup/independent/install-kubeadm/
• https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/
• https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/
• https://kubernetes.io/docs/tasks/tls/certificate-rotation/
• https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/
• https://kubernetes.io/docs/reference/access-authn-authz/bootstrap-tokens/
• https://medium.com/@toddrosner/kubernetes-tls-bootstrapping-cf203776abc7

Originally published at https://supergiant.io.