3 key elements to protect a Kubernetes cluster

Published in

Mirantis

5 min readJun 1, 2022

Kubernetes changed how we structure, deploy, and run our applications and became a de-facto standard for running infrastructure at scale. With the rapid adoption of container-based technologies, organizations are increasingly concerned about the security of their Kubernetes clusters. And they should be! While cloud and enterprise distributions provide solid security capabilities, they require tuning according to match organizational security needs.

Note: This blog post was originally published by HelpNetSecutiry. You can view the original post here.

This article is a summary of a talk that I gave at CloudWorld 2022. If you want to learn more about the topic, you can watch the talk here, access slides here, and code snippet here.

In this article, I’ll go over three fundamental areas you need to consider to protect a Kubernetes cluster:

Role-based access control (RBAC)
Open Policy Agent (OPA)
Network policies

Let’s dive in!

Role-based access control

Let’s consider an organization with three application teams (Blue, Green, and Red). Because these teams are working on different products, they should be given different access to the Kubernetes cluster. For example, the Green and Red teams should not be able to see, access, or delete what the Blue team deployed.

RBAC is a way to control what Kubernetes resources users can access. While RBAC is enabled by default, you must configure it to use it.

There are 5 key elements to RBAC:

Subjects — Users and processes
Resources — Objects to which access should be restricted
Verbs — Sets of operations that can be executed, often referred to as actions
Roles — An object that connects API Resources with Verbs
RoleBinding — An object that connects Roles with Subjects

Let’s go back to our organization example and define a policy where only the Blue team can create, delete and list Pods, Deployments, and Services.

First, we create a Role object named `role-blue`, where we define the actions that can be performed on specific Kubernetes resources. In this specific case, the Role enables the actions `create`, `delete`, and `list` to be performed on the resources; `pods`, `deployments`, and `services`.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: blue-role
  namespace: blue
rules:
- apiGroups: ["", "apps"]
  resources: ["pods", "deployments", "services"]
  verbs: ["create", "delete", "list"]

Next, we create a RoleBinding named `blue-rb`. This RoleBinding belongs to `blue-ns`, which links the above-created role `role-blue` with the blue team, named `blue`.

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: blue-rb
  namespace: blue-ns
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: blue
roleRef:
  kind: Role
  name: blue-role
  apiGroup: rbac.authorization.k8s.io

Once these resources are applied to the cluster, the users from the `blue` team will have the ability to perform the operations defined in `role-blue`.

Open Policy Agent

Open Policy Agent (OPA) is a general-purpose policy engine that unifies policy enforcement across the stack. Its high-level declarative language provides flexibility to specify policy as code. You can use OPA to enforce policies in Kubernetes, CI/CD pipelines, API gateways, and more. Let’s dive into how to use and implement it in Kubernetes.

The Kubernetes implementation of OPA is called Gatekeeper. It is designed and deployed as an AdmissionController that intercepts requests, processes them, and responds back with a permit or deny response.

When permitted, the object gets deployed on the cluster; otherwise, the request will be rejected and feedback provided to the user. Administrators can define policies that instruct Kubernetes to limit the resources such as memory or CPU that containers or namespaces can consume, approve only containers based on images from specific registries, restrict NodePort service creation, or enforce standard naming.

For example, here is a sample template and constraint policy that allows creation of Pods only when ResourceQuota is configured in a namespace.

constraint.yaml

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredResourceQuota
metadata:
  name: namespace-must-have-resourcequota
spec:
  match:
    kinds:
      - apiGroups: ["", "apps"]
        kinds: ["Pod"]
    excludedNamespaces:
      - gatekeeper-system

template.yaml

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8srequiredresourcequota
  annotations:
  description: Requires Pods to launch in Namespaces with ResourceQuotas.
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredResourceQuota
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredresourcequota
        violation[{"msg": msg}] {
          input.review.object.kind = ["Pod"]
          requestns := input.review.object.metadata.namespace
          not data.inventory.namespace[requestns]["v1"]["ResourceQuota"]
          msg := sprintf("container <%v> could not be created because the <%v> namespace does not have ResourceQuotas defined", [input.review.object.metadata.name,input.review.object.metadata.namespace])
        }

Network policy

Network policies are very similar to a regular firewall but differ in the sense that they are application-centric. When you define a network policy for an application, Kubernetes automatically applies those rules to the associated containers due to the highly dynamic environment in which containers are constantly created and terminated. Network policies control the flow of traffic to or from these containers.

By default, network traffic to and from Pods is not restricted. A good starting point is to set a deny-all traffic rule, and then only allow necessary traffic.

By default, Kubernetes uses a flat network structure allowing any Pod to communicate with other Pods or Services in the cluster. In a cluster with multiple applications or multi-level applications, defense-in-depth plays a key role in securing the communication layer. Network policies allow us to achieve that.

Here is an `app1-network-policy` that applies the following rules in the `blue` namespace for the Pods with label “role=db”:

[Ingress] Allows connections from the ipBlock 172.17.0.0/24 on port 6379
[Ingress] Allows connections from other pods if they are labeled as “role=frontend” and if they belong to the namespace with labels “project=myproject” on port 6379
[Egress] The pod can communicate with other pods with IP range 10.0.0.0/24 on port 5978.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: app1-network-policy
  namespace: blue
spec:
  podSelector:
    matchLabels:
      role: db
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - ipBlock:
        cidr: 172.17.0.0/24
    - namespaceSelector:
        matchLabels:
          project: myproject
    - podSelector:
        matchLabels:
          role: frontend
    ports:
    - protocol: TCP
      port: 6379
  egress:
  - to:
    - ipBlock:
        cidr: 10.0.0.0/24
    ports:
    - protocol: TCP
      port: 5978

Conclusion

With RBAC, OPA, and network policies in place, you can protect your Kubernetes cluster by assuring that contributors have the proper access, that security policies are enforced, and that the network is tightly secured.

About Mirantis

Mirantis helps organizations ship code faster on public and private clouds. The company provides a public cloud experience on any infrastructure from the data center to the edge. With Lens and the Mirantis Cloud-Native Platform, Mirantis empowers a new breed of Kubernetes developers by removing infrastructure and operations complexity and providing one cohesive cloud experience for complete app and DevOps portability, a single pane of glass, and automated full-stack lifecycle management with continuous updates.

For more details, visit https://mirantis.com