Secret to securing your Kubernetes networking in 9 steps

Configuring Kubernetes security for secure communication within the cluster can be very confusing. Julia Evans on her blogpost about this:

The various Kubernetes components have a TON of different places where you can put in a certificate/certificate authority. When we were setting up a cluster I felt like there were like 10 billion different command line arguments for certificates and keys and certificate authorities and I didn’t understand how they all fit together.

So what components exist within a cluster relevant to this discussion?

Kubernetes cluster components include a Master node which has the following components:

• Kube api server — responsible for the overall functioning of the cluster and interacting with the user. This is basically the control center of your cluster
• etcd — storage for your kubernetes certificates and maintains state for your cluster.
• kube-scheduler — responsible for scheduling decisions related to the pods
• kube-controller-manager — manages all the controllers

Worker node:

• kubelet- responsible for ensuring that the workloads are running as they should be running
• kubedns — responsible for setting up things like dns addresses for your kubernetes services
• kube-proxy — responsible for configuring networking components like iptables for service routing
• Pods and containers — your actual workloads

We will see how the different parts communicating with each other are secure.

So what communications should I be concerned about?

Out of all of the above, the API service, kubelet and etcd are the most critical components. Let’s say the pod needed a certificate, it would talk to the kubelet and then the kubelet would communicate with the API server to get access to the certificate stored in etcd.

All communications with the master happen through the API server. Similarly all communications with the worker node happen through the kubelet. Etcd holds all the important information for your cluster. So anything communicating with etcd should be well protected.

So let’s pick up the cases one by one.

Case 1: Communication from API server to kubelet

The API server communicates with kubelet for things like fetching logs, takes in configuration from the user and communicates with the kubelet to make it a reality.

By default, the communication from API server to kubelet is over unauthenticated TLS. This makes it unsafe to run the cluster over untrusted networks since this communication is susceptible to man (or woman) in the middle attacks. If you’re running things on your private network, this behaviour should be acceptable. If you want, however, you can use the flag “kubelet-certificate-authority” to verify kubelet’s server certificate. Note SSH tunneling is deprecated so that is no longer a recommended solution.

Case 2: Communication from kubelet to API server

This would happen in cases such as when a new node comes online and that node will need a new certificate to communicate with the API server. The API server has a Kubernetes API that lets get TLS certs signed by a CA that you can control.

Requests are using mutual TLS and API server listens on port 443. So by default, the master can run on untrusted networks.

Another thing to note here is that you will need to setup certificate rotation for your kubelet certs. By default, they are issued with a 1 year expiration date.

kubeadm alpha certs check-expiration

The command for kubelet to rotate certificates is --rotate-certificates

Case 3: Communication from user/admin to API server

The admin interacts with the API server on various instances to manage the cluster.

The authentication method for this communication depends on your needs. There are several options such as:

• Static username password.
• Authenticating proxy
• x509 client certificates
• OAuth tokens

Case 4a: Communication from API server to etcd

This might happen in cases where a specific node or pod asks for a certificate from the kubelet, the kubelet makes a request to the API server and then the API server fetches it from etcd.

The communication with etcd happens over localhost on port 80 and is not authenticated or encrypted. You should setup TLS for this part of the communication to verify identity of the etcd instance with the etcd server by specifying the flag “etcd-certfile”.

In the reverse direction, the API server listens on the secure HTTPS port 443.

Case 4b: Communication from etcd to API server

The API server listens on the secure HTTPS port 443.

Case 5: Communication between two instances of etcd

Etcd instances talk to each other to keep state in sync.

Instances of etcd communicate over mutual TLS already so this part of the cluster should also be safe to run on untrusted networks.

Case 6: Between two pods

As you know, pods in different or same nodes communicating seamlessly with each other without NATing is one of the foundational principles of Kubernetes networking. So this is another important path of communication.

Currently this communication is not encrypted and not authenticated. There are ways to make this more secure such as using some network policy provider like Tigera’s Calico or Cilium to protect service to service traffic. At the very least, it is highly recommended to use some network policy provider based on your application.

You can also use a service mesh like Istio to enforce mutual TLS for traffic between services. Using a whole service mesh just to enforce mutual TLS between service to service communication might seem like a huge ask, so maybe use a network policy provider that provides the same without needing a proxy in front of every pod.

One obvious thing to recommend here is Pod Security Policies. Will address this more in a separate blog post.

This communication path is so critical this probably requires a whole post of its own.

Case 7: From pods to API server

This type of communication is done over plain HTTP but really this type of communication shouldn’t even be happening directly.

Case 8: Other components like kubescheduler, kube-controller-manager talking to the API server.

This happens over localhost and you can setup secure communications here as well. Use

--secure-port

for securing communication from the kube-controller-manager to API server. This uses port 10259 by default and uses HTTPS authentication. You can also set the option and many others:

--tls-version-min

Starting Kubernetes 1.13, secure port 10259 can be used for secure connections from kube-scheduler to api server.

Case 9: Between two nodes

You can use the same certificate signing API if you want to ensure node to node communication also uses certificates.

Conclusion

Summarizing all of this information in a table for you:

If you found any errors with the above, would love to rectify them, or if you any feedback, would love to hear more!