Security as standard in the land of Kubernetes.

Chris Cooney
Apr 5 · 9 min read

Kubernetes will automatically handle high availability, deployments, configuration and secrets for you, but it won’t enforce security on its own. That takes some work.

That’s not to say that Kubernetes is insecure, it is simply a different way of approaching software. A way that favours continuous delivery, zero downtime deployments and high availability. Kubernetes will do these things for you, with minimal effort. Security, on the other hand, requires voluntary effort. This has been my job for the past few months and I thought I’d share the lessons I’ve learned. First, the overarching ideal we had in mind.

Security as Standard

This is simple. It should not be the effort of every single engineer to make sure every layer of their application, deployment and configuration is secure. Think about the amount of rework involved in this.

  • The silo this creates and the disjointed architecture you’re landed with, because everyone has a slightly different method based on their experience.

So How Do We Achieve This?

This will form the meat of this article. I have compiled a list of tools and practices that one can use to help foster a technical ecosystem that creates a safe, secure environment for engineers to operate in.

The Kubernetes NetworkPolicy

I have no doubt that the majority of people reading this article will know all about the NetworkPolicy in Kubernetes. When you’re starting out with security in a Kubernetes cluster, this is the gold. In short, it allows the engineer to lock down which other services can talk to this pod. For example, you can create a NetworkPolicy like this:

kind: NetworkPolicy
name: api-allow
app: my-app
- from:
- podSelector:
app: my-other-app

Utilise Helm for Consistency

Helm allows you to bundle up lots of Kubernetes resources into a single chart. Its most common use case is to make the deployment of 3rd party software trivial — you can peruse the huge collection of stable helm charts at your leisure. Everyone who’s anyone has got a chart on there and it should be your first port of call when you want to make some open source tooling available in your kubernetes cluster.

Role Based Access Control (RBAC) for Services

RBAC is a very common feature in Kubernetes cluster. I would strongly advise turning it on. We typically use it for third party applications and developer access, but we can also use it for our software, in the form of ServiceAccount resources. We can link these to a specific Role or ClusterRole and hey presto, no more risk of applications going rogue inside your cluster.

Validate Your Yaml Before Deploying using Kubescore

So you’ve got a badass helm chart for your applications, but not everything is an application. There are plenty of things, Nexus servers and CI tools, that won’t be deployed through the typical means. Often, it’ll be a cheeky kubectl command from an engineer’s machine that opens up the widest vulnerability.

Test your Cluster Configuration with kube-bench

There is a brilliant tool called kube-bench that will analyse the configuration of your nodes. This will test out things like, whether you have disabled privileged containers and that your kubelet is communicating with the master nodes in HTTPS.

Create a Standard Set of Base Docker Images

One gaping hole with all of this is the application containers themselves. Vulnerabilities creep into those all the time and this unknown quantity creates a wonderful little hiding place for difficult to diagnose bugs and vulnerabilities.

And look out for the latest tag

Ensure all docker images that are deployed into the cluster are pinned to a version. The latest tag is a dangerous game indeed — you have absolutely no idea what software is running after each pod recycle.

Istio Mutual TLS

If you’ve been looking into Istio, you’ll know that it very recently announced v1.0.0. That’s it folks, they’re ready for production. Istio offers an absolute plethora of monitoring, tracing and resilience. It also has an awesome feature for enabling mutual TLS between Istio managed applications. If your application has an envoy proxy, you’re in the money.

But what is Mutual TLS?

Mutual TLS is a little like HTTPS. It involves one party verifying the existence of the other and, subsequently, establishing an encrypted communications channel. However, HTTPS is predicated on the client verifying the server. In mutual TLS, two clients will verify each other.

Does it impact the applications?

This is the cool bit. Istio Mutual TLS happens outside of the application, inside the envoy proxy that is coupled with your applications. Your apps can send a request in HTTP and Istio will silently verify the source and encrypt the traffic for you. Developer effort? Absolutely nothing. Security benefits? Strong cryptographic encryption in transit within the cluster and a massive blocker to man in the middle (MITM) attacks. Not bad at all.

Controversial: Ban the LoadBalancer and NodePort Service Types — Stick to ClusterIP

This one tends to split the room a little. We see the LoadBalancer service type crop up everywhere in services. It is often the default configuration for many helm charts that are seeking to allow ingress traffic from outside the cluster. There are some serious drawbacks to this:

  • Each Load Balancer is going to talk to your nodes on a different port. The more ports you’ve got open, the greater the attack surface.
  • Who the hell knows what configuration is going on between those load balancers and their target nodes. It is totally at the discretion of whatever controller you’ve installed.

So what can we do instead?

Create a single way in. One Load Balancer. For us, we use an AWS ALB, but you can use an NLB or a classic ELB if the mood takes you. The aim of the game is to create only one route and one set of ports that traffic flows in on. This minimises your attack surface. With all of your applications running as ClusterIP instances, no additional port exposure is needed and the routing is done for you by Kubernetes. The only thing left to do is to pick a single ingress controller to run as a NodePort, perhaps NGINX or Traefik, and hey presto! You’ve got simple networking with a tiny attack surface.

Very Controversial: Take control of the build and deployment process

I’ve included this one as an opinion piece. It is much more controversial than the others that I have discussed. Teams often enjoy owning their own Continuous Delivery (CD) pipelines. This is something they will need to work with on a daily basis. There are classic disagreements amongst engineers about which CI/CD tool to use.

And this opens an attack vector, a big one.

Until you know, not think or guess, but know that your engineers are making use of the automation that has been put in place, all of this effort could be cancelled out by a single dodgy config. Your helm chart might wrap services in steel wool and spray it with holy water, but it’s no good if it isn’t being used.

But a word of warning…

Think long and hard about this one. You’re taking on a huge maintenance and reliability burden. If that CI/CD tool that you’ve built is flakey, it impacts every developer, immediately. It is a single point of failure and your life will be spent fighting fires. Think hard before shouldering this.

Often trust and education is the best solution…

Short of taking over the CI/CD process, the best thing you can do is be involved in the creation of each team’s set of tools. Answer questions, talk about capabilities. Make yourself available to give advice to teams. Give people autonomy and responsibility over their own solution by letting them know what good looks like. This approach doesn’t offer guarantees, but it presents a much more manageable operational overhead.

Okay, I’m Fresh Out.

I’ve avoided discussing specific networking techniques that we have employed and I’ve ducked around how we can write our apps in a more secure way. The trick here has been to get some quick wins that will seriously narrow the attack surface of any Kubernetes cluster while requiring very little effort from your software engineers.

This is no longer updated. Go to instead

Chris Cooney

Written by

Software Engineer.

This is no longer updated. Go to instead