Securing Kubernetes on Microsoft Azure: are your container doors wide open?

Matt Asay from InfoWorld wrote the following a couple of weeks ago: “The operating system no longer really matters. And for developers and the cloud, that means that Linux no longer really matters. We now live in a Kubernetes world.”

Kubernetes is quickly becoming the new standard for deploying and managing software in the cloud. Microsoft last year launched their managed Kubernetes offering in Azure called Azure Kubernetes Service (AKS) and IBM’s main driver behind the $34 billion acquisition of Red Hat in 2018 was its Kubernetes-based OpenShift offering.

Few people have extensive experience with Kubernetes. And if they did have training or have gotten their hands dirty with deployments, how many only focus on general engineering and administration and overlook the security aspect?

A quick search on Shodan, the leading search engine for finding vulnerable internet-connected systems, reveals that in early 2018 over 15K+ Kubernetes dashboards were directly facing the internet, and few of them had security measures in place.

Have you secured your Kubernetes environment, or are your container doors wide open?

Securing Linux to secure Kubernetes

Jose Alvarez summarizes it nicely: you need to make sure you tackle the basics first — secure Linux to secure Kubernetes. There are a couple of things to look at:

  • AppArmor is a Linux kernel security module that supplements the standard Linux user- and group-based permissions to confine programs to a limited set of resources. AppArmor can be configured for any application to reduce its potential attack surface and provide greater in-depth defense.
  • By default, the administrator can SSH to any deployed node in a Kubernetes cluster. Consider disabling SSH access to the cluster nodes.
  • Ensure that Docker itself is configured per security best-practices. Here’s an article on Docker security best-practices.
  • Ensure that your underlying hosts are hardened and secure. I recommend the CIS benchmarks as a starting point. The fine folks at Aqua Security also open-sourced an automated checker based on CIS recommendations. Check it out: kube-bench.

By no means is this a complete list, but a justa few things to get you thinking on this topic. Do you have other things you secure on Linux as part of hardening your Kubernetes environment? Please share them in the comments below.

Whilst it’s possible to run Microsoft Windows Server as the OS for Kubernetes worker nodes, more often than not, the control plane and worker nodes will run a variant of the Linux operating system. There might be many factors that govern the choice of Linux distribution to use (commercials, in-house skills, OS maturity), but if it’s possible, use a minimal distribution that has been designed just for the purpose of running containers.

Examples include CoreOS Container Linux, Ubuntu Core, and the Atomic Host variants. These operating systems have been stripped down to the bare minimum to facilitate running containers at scale, and therefore, have a significantly reduced attack surface.

PRO TIP: Aqua Security is also available directly from your Kubernetes cluster overview page in Azure!

Kubernetes is a framework, not a platform

So why is Kubernetes not secure by default? Jussi Nummelin summarizes it nicely: from a security standpoint, Kubernetes is more like a framework intended to be utilized in building more higher-level solutions. What that means in practice is that plain Kubernetes does not lock down everything properly by default, instead it is expected to be configured by the operators setting it up.

Not sure how Kubernetes works? Microsoft together with the Cloud Native Computing Foundation (CNCF) released a cartoon called “Phippy Goes To The Zoo” telling about the basics. But also Daniel Sanche has written a great overview of the core concepts in Kubernetes you could look at.

Tesla found out, the hard way

I wrote about the rise of the cryptocurrency miners in a previous blog. Well, add Tesla to the legion of organizations that have been infected.

In a report published early 2018 researchers at security firm RedLock said hackers accessed one of Tesla’s Amazon cloud accounts and used it to run currency-mining software. The initial point of entry for the Tesla cloud breach, the report said, was an unsecured administrative console for Kubernetes.

“The hackers had infiltrated Tesla’s Kubernetes console which was not password protected,” RedLock researchers wrote. “Within one Kubernetes pod, access credentials were exposed to Tesla’s AWS environment which contained an Amazon S3 bucket that had sensitive data such as telemetry.”

ArsTechnica has a great in-depth article on the breach which you can read here.

Netscylla was intrigued on how easy this could happen to any customer cloud estate. So, they decided to do some OSINT (Open Source reconnaissance) and their own research. Want to understand what ports are involved and how to find vulnerabilities? Have a look at their writeup.

Kubernetes on Microsoft Azure

One of the questions I have been hearing a lot from customers, is about how RedHat’s OpenShift (which is based on Kubernetes) compares to the managed Kubernetes services provided by most of the major public cloud providers.

Google introduced the world to Kubernetes mid-2014, and their Google Cloud Platform has a managed service called “Google Kubernetes Engine” (GKE). Amazon has a service called “Amazon Elastic Container Service for Kubernetes” (EKS) and Microsoft has a managed Kubernetes based offering called “Azure Kubernetes Service” (AKS).

If you want more detail about choosing OpenShift or not, Chris Saunders, who works at RedHat, wrote a lengthy article on what OpenShift provides additionally to a plain-vanilla Kubernetes environment. For now, I’ll focus on Microsoft’s Azure Kubernetes Service.

Amplifying that Microsoft is serious about containers, Kubernetes and open source in general is the hire of Brendan Burns. Brendan helped create the Kubernetes open source container orchestration technology and joined Microsoft to work on the Azure in 2016.

Want to get your own Kubernetes cluster running on Microsoft Azure? Microsoft has great documentation on this, and how to get your first application running on that cluster.

Securing your Kubernetes environment

Here are some general recommendations on improving security:

  • Regardless of the network policy in place, use multi-factor authentication for all access

Etienne Dilocker has written a lengthy article on comparing authentication methods for Kubernetes.

  • Apply strict controls to network access, especially for UI and API ports

Microsoft provides an own network policy module to implement Kubernetes network policies with the Azure CNI plugin for acs-engine and AKS called Azure NPM. My good friend Daniel Neumann wrote a blog on how to use this, and you can find the relevant Microsoft documentation here.

  • Use SSL for all servers and use valid certificates with proper expiration and enforcement policies
  • Deploy appropriate network protections like reverse proxy to enable connection to sensitive servers

John Kindervag’s Zero Trust Network Architecture is a great concept to embrace, and optimizes the security architectures for future flexibility. It works great in a data-centric world with shifting threats and perimeters and provides you with new network designs that integrate connectivity, transport, and security around potentially toxic data. John calls this “designing from the inside out.”

  • Build a pod security policy by preventing pods from running as root, as well as from accessing host ports and certain volume types
  • Configure your Kubernetes pods to run read-only file systems
  • Restrict privilege escalation in Kubernetes with role-based access control
PRO TIP: Docker is often viewed as a simple and fast replacement for virtualization. While this is true to certain extent it should be noted that containers, compared to virtual machines, do not provide the same level of isolation from either the host or other containers running within the same host. Please always remember that the host’s kernel is shared among all the running containers.

The Cloud Native Computing Foundation (CNCF) published a couple of helpful pointers: 9 Kubernetes Security Best Practices Everyone Must Follow and also Nikita Mazur shared some great insights into the things you should be doing to secure your Kubernetes environment.

Beware: the Kubernetes network stack!

The most contentious point for building a Kubernetes cluster is the network stack. There are a ton of providers available and if you have never deployed K8s before, it can be daunting!

Kubernetes handles networking using a different approach to the normal ‘Docker way’ of doing things. Docker uses port-forwarding from the host to the underlying container using virtual bridge network. This means that coordinating ports across multiple devices is a difficult thing to do at scale and exposes users to cluster-level issues outside of their control.

Kubernetes on the other hand imposes the following requirements on its network implementation:

  • All Pods should get their own IP Address, and the container should be able to see itself as the same IP that others see it
  • All containers can communicate with each other without NAT
  • All nodes can communicate with all containers (and vice-versa) without NAT

Rory Chatterton does a great job in explaining, and providing you with an approach to use.

DevSecOps is the new DevOps

While DevOps approach to software delivery has positively revolutionized the IT industry, the security requirements have largely been left unattended. This is what DevSecOps aims to fix.

DevSecOps is one of the most important DevOps trends. It is an approach to IT operations security, allowing to utilize the principles and best practices of DevOps to ensure better, faster and more secure software delivery. This essentially means, that all the security requirements are codified and weaved into the automated unit tests from the start, instead of having to deal with them before the product release.

“DevSecOps is the trend to automate all security checks by codifying them in the unit tests and using them from the very beginning of the software development, not at the end of the cycle.”

Arseny Chernov has written a lengthy article on securing DevOps and what approach to take. A recommended reading if you want to go from DevOps to DevSecOps.

Summary

Kubernetes is a very powerful platform that makes application deployment much easier compared to a traditional on-premises server or a VM. That’s probably one of the key success factors that made Kubernetes adoption boom over the last couple of years. The counterpart is that this way of doing things is relatively new, which unfortunately sometimes leads to poor implementation of good practices in terms of security.

The need to secure your Kubernetes environment is real, as exploits are now being found in the wild. Learning about Kubernetes and understanding how the framework works is essential to understand what to secure and I would certainly recommend Nigel Poulton’s course for this.

— Maarten Goet, MVP & RD