Kubernetes Security — Part I

Complexity is the worst enemy of Security

- Bruce Schneier

Kubernetes is designed to be highly portable, with multiple deployment options like bare metal, on-premise, public cloud, and managed service making it easier for migration of workloads. It is designed out of the box to be customizable and the end users are expected to turn on certain functionality to secure their cluster. All of the customization options that are available for Kubernetes make the platform relevant for large-scale use cases and a variety of scenarios but this also means that the engineers responsible for deploying the Kubernetes platform need to know about all the potential attack vectors and vulnerabilities that some poor configuration can lead to.

Moreover, you may have noticed that Kubernetes natively does offer some built-in security tools and a handful of security features that helps in some situations, but makes it exceedingly difficult to secure every layer of a Kubernetes environment.

On the contrary, the fact that Kubernetes is such a platform where you could find so many integrations and solutions that you could actually make it easy to build an automated, systematic set of processes that puts security into the core of the Kubernetes build and deployment process. This would enable a tightly integrated security strategy that mitigates threats at all layers and levels of your stack. This article would be focusing on securing Kubernetes at a cluster level.

Securing Kubernetes hosts:

All security typically would start at the infra level. There are a few important checks that we could do here. Like:

• Standardizing the use of a secure enterprise OS with the latest version across the cluster.
• Reducing the surface area of attack by hardening the OS
• Ensure all necessary patches are implemented
• Automate your configuration management system
• Implement all necessary firewall rules
• Ensure all other basic security measures specific to your data center.
• Always ensure you are running the latest version of Kubernetes.
• Use features like rolling updates, and node pool migrations that help you complete an update with minimal disruption and downtime.
• Limit SSH access to nodes in the cluster, and make use of “kubectl exec” command, which will provide direct access to the container environment instead of accessing the host.

Securing access to Kubernetes API

The Kubernetes platform is entirely API-driven, therefore controlling who has access and what actions they are allowed to perform is our first line of defense. There are 3 things we can do at this level:

Secure all API traffic:

All API communication in the Kubernetes cluster should be encrypted by TLS by default. Generally, when you install Kubernetes, the installation itself should create all the necessary certificates and ensure they are distributed to all cluster components. Of course, It is important that we familiarise ourselves with the setting of all components to identify any potential insecure traffic but it’s recommended to make use of more modern network technologies like Service Mesh, which can enable TLS by default but also provide telemetry information about the network traffic.

API Authentication:

All actions performed by the Kubernetes API client must be authenticated. Authentication can be performed by either making use of x509 client certificates or by making use of service accounts.

Kubernetes provides a number of built-in mechanisms for API server authentication. We should choose an authentication mechanism that fits better with our own use case, eg.

If you are using small clusters only for R&D, workshops, non-prod environments, etc, you could use

1. Static Token files make use of clear text tokens stored in a CSV file on the API server node. But do note that any modifications in these credentials that are stored in this file would require the API server to be restarted in order for the changes to be effective.
2. X509 Client Certs are also available and are mostly configured by default during the installation process but are considered unsuitable for production use, as Kubernetes by default, does not support certificate revocation which means that the user credentials cannot be modified or revoked without rotating the root certificate authority key and then re-issuing all cluster certificates.
3. Service Accounts Tokens are a common mechanism for authentication and their primary intended use is to allow workloads running in the cluster to authenticate to the API server, however, they can also be used for user authentication.

Or if you are working with a large production cluster, then it would make sense to integrate your existing OIDC or LDAP servers by externalizing authentication by using short-lived tokens and leveraging centralized groups for authorization.

Some managed Kubernetes distributions like GKE, EKS, ROSA, and ARO also support authentication using credentials from their respective IAM providers.

API Authorization:

After authentication, the next step is an authorization which ensures that once you have logged in, you are granted only the permissions that you need to perform certain actions. These permissions combine verbs (get, create, delete) with resources (pods, services, nodes) and can be namespace-scoped or cluster-scoped.

Kubernetes comes with an integrated RBAC (Role Based Access Control) which does provide some out-of-the-box roles which might basically fit well for most common use cases and at the same time provide a reasonable level of security. It’s still recommended to create your roles with more fine-grained permissions following the principles of least privilege. Just as I mentioned with authentication, simple and broad roles might be appropriate with small clusters, but with larger clusters, it is necessary to separate teams into separate namespaces with more limited roles. This not only helps with security but also management and governance when an escalation does happen.

Kubernetes authorizes API requests using the API server. It evaluates all of the request attributes against all policies and allows or denies the request. All parts of an API request must be allowed by some policy in order to proceed. This means that permissions are denied by default.

Securing Kubelet service:

An agent that runs on each node in the cluster. It makes sure that containers are running in a Pod. The Kubelet takes a set of PodSpecs that are provided through various mechanisms and ensures that the containers described in those PodSpecs are running and healthy.

By default, Kubelets allow unauthenticated access to all its HTTPS endpoints. We should enable authentication and authorization for the Kubelet as well.