Securing your Kubernetes Deployment
Written by Chandni Vaya. Peer reviewed and contribution by Haytham Elkhoja.
In case you haven’t heard, Kubernetes is a big deal. It allows orchestrating the deployment and lifecycle of your applications running on containers.
Security is always a point of concern when adopting new platforms and technologies. Here, we try to shed the light on some of the best practices and things to be aware of using Kube for your modern distributed app.
And depending on what Kubernetes platform you’re using, a cloud-based managed Kubernetes service like IBM Cloud Kubernetes Service, Amazon EKS, Microsoft AKS or Google GCP, or an on-prem Kubernetes solution such as IBM Cloud Private, Red Hat OpenShift Container Platform, Pivotal Kubernetes Service or Rancher, developers, SREs and sysadmins alike should be aware of the different attack vectors of a Kube architecture.
Kubernetes Attack Vectors
Attack vectors are means or gates by which malicious users will try to crack the armor and attack your platform in the following areas:
- Container Images
- Platform and Worker Nodes
- Communication
1. Container Images
The app (or container image) is where developers will eventually have the most control over security. And nothing changes much in terms of application development when applying security best practices.
But Docker images are made out of base images and tens of layers that are built by third-parties, developers need to feel confident in the authenticity and security of the container images layers. Organizations, on the other hand, need to enforce compliance rules and proper processes when re-using third-party base images.
For example, a normal Java application needs JDK as a base image, which in turn needs a base OS image that could have security vulnerabilities, or that certain version of JDK base image might not be vetted by the organization for production usage.
Thankfully, many tools can help you assess the security of your base images and enforce compliance rules. IBM Vulnerability Advisor and open-source tools such as Twistlock, CoreOS Clair, Dagda, Anchore, etc… continuously test your image for possible vulnerabilities and alert you when developers are not using proper base images.
Other best practices when building your Docker images:
- Practice the principle of least privilege — do not give your application any type of root access.
- Go for minimal and official base images — more often than not — the open source community or vendors publish usable and signed minimal official images that should be verified to avoid MITM attacks. Make it a habit to verify images as you pull them by using:
export DOCKER_CONTENT_TRUST=1- Keep your Docker file and image clean of sensitive information— this means never including tokens, IP addresses, user/passwords etc.. Bootstrap them when running instead.
2. Platform and Worker Nodes
Now depending on the solution you’re using, you might or might not have access to any of the nodes. If you do have admin privilege on the worker nodes or are deploying an on-prem Kube solutions, then you should consult your vendor documentation on security best practices. For example, IBM’s Cloud Private has guidelines on hardening and securing IBM Cloud Private for production. There are also guides to follow to comply with PCI DSS. Red Hat’s OpenShift Container Platform has also published documentation on how to harden and secure it.
Here are some best practices on securing your Kubernetes platform:
- There are plenty of OS security hardening articles online and benchmarks from organizations like CIS (Center of Internet Security) that guide administrators and security staff on how to secure their environments.
- Harden your Docker Engine or containerD runtime. containerD is still relatively new (as of April 2019), but is quickly replacing Docker engine as main runtime in many on-prem Kube solutions.
- Encrypt the worker nodes file system with solutions like LUKS or dm-crypt.
- In-memory encryption and the use of runtimes enclaves is growing amongst cloud providers alongside encryption at rest and in motion. You may have heard of Fortanix or Scone, which are some of the solutions that rely on Intel SGX to make that magic happen.
- Enable secrets data encryption by using Key Management Service (KMS) providers and plugins. Most on-prem solutions have plugins and documentation to integrate HashiCorp’s Vault for example.
- Authentication and access control policies become an issue when many users are sharing a cluster. Most Kubernetes can plugin to off-the-shelf Identity and Access Management tools (IAM) for Role Based Access Control (RBAC) and fine-graining control base on namespaces for example.
3. Communication
If you’re on Kubernetes that means that you’re deploying distributed systems and unlike monolithic applications that rely on in-process messages, in a microservices world, messages use the network.
Before we dive in securing communication on Kubernetes we have to differentiate two things: Control-plane and Data-plane.
Control-plane communication is basically all the traffic that goes between Kubernetes cluster management daemons and the Kubernetes API Server and that’s normally secured via HTTPS.
At its simplest form, encrypting all communication traffic within the cluster should be a non-brainer. Both IBM Cloud Private and Red Hat OpenShift Container Platform provide documentation to enable end to end IPSec between hosts.
However, one thing to note here is that all Kubernetes control-plane communication normally passes through HTTPS, so using IPSec encrypts traffic a second time could be seen as overhead in some situations but valuable for FIPS compliance for example.
Data-plane communication is basically all the traffic that goes between pods and services that host the containers. Now, you could manually deploy SSL certificates in Ingress rules and that’s fine, but also tiresome.
The alternative would be to automate that process using Mutual TLS from Istio. While Istio is a service mesh and much more than simply an encryption tool, Istio’s mTLS guarantees that communication between services created for a deployment are encrypted. Istio makes it that clients and servers verify the authenticity of the certificates in both directions.
When talking about Data-planes, Kubernetes allows you to micro-segment and assign policies to how pods talk to each other and to other nodes within a cluster. By default, pods are non-isolated as in they accept traffic from any source which is problematic in a shared environment or when deploying multiple applications in a single cluster.
NetworkPolicies aren’t native to Kubernetes, they are however applied by a running Software-Defined Networking component such as Calico, OVS or VMware NSX-t.
Above are some of the important security measures that should be taken into consideration without fail because be certain that security attacks are imminent and security shouldn’t be an after-thought, but a primary concern when modernizing your app.
