Open Source Tool to Establish Least Privileged Best Practice for Kubernetes Clusters

With New Kubernetes Plugin, Role-Based Access Control Has Never Been Easier

If you are using or thinking about using Kubernetes (K8s), you’re going to want to think about how to minimize its attack surface. We’ve seen containers and Kubernetes, itself, being exploited by attackers. For example, an attacker used an administrative console that wasn’t password protected to gain admin privileges for a Tesla K8s cluster — once in, they had access to everything, including credentials to a cloud environment that allowed them to launch a pretty sophisticated cryptojacking operation.

So, what can you do to protect yourself? A good start is to follow the principles of least privilege when setting up your Kubernetes cluster to make sure users and services only have the access they should. To help you achieve least privileged access permissions, we’ve developed an open-source tool, Kubectl-RBAC. This tool makes it quick and easy to establish and maintain role-based access control (RBAC) for your K8s clusters.

Defining Role-Based Access Control for K8s Clusters

In Kubernetes 1.8, RBAC was supported. The trick has been implementing it. This new open source RBAC tool makes it easy for you to dynamically configure policies and authorizations for every user and service account, using the Kubernetes API, within your K8s’ clusters. For those of you interested in the full Kubernetes documentation, you can find it here, for everyone else, here’s a quick description of how RBAC works in a K8s environment. In general, the RBAC distinguishes between “Users” and “Service Accounts.”

· Users — developers, IT managers and any other person that needs to perform operations in the K8s cluster

· Service Accounts — processes inside pods that perform operations in the K8s cluster

Before RBAC was introduced, the default permissions for a K8s pod was “*”/admin. This meant that if a pod was breached an attacker could then take over the whole cluster and cloud environment. Now, with RBAC, permissions can be defined for every pod, with regards to which actions it can perform, so exploits can be contained.

Permissions are defined for each “Role.” Each “User” is then assigned a “Role,” using either “RoleBinding” or “ClusterRoleBinding.” The difference between the two is that “RoleBinding” is effective within a particular namespace, while “ClusterRoleBinding” is effective cluster-wide.

The Goal of RBAC for K8s

When you look at security within K8s, you can really break it down to two main concerns:

1) Securely deploying and configuring the K8s cluster, itself.

2) Securely deploying and configuring the applications running within K8s.

RBAC is part of application security and is a must for any environment. Within K8s, it typically falls to the application developer to configure RBAC properly because cloud-vendors don’t know the specifics of the app, so they can’t take responsibility for configuring it properly.

When configuring, one of the primary considerations is to make sure if one pod within a cluster is compromised, say due to an application bug, it can’t be used to take over the whole cluster and give an attacker access to everything (database/users/etc.). With RBAC, you can specify very fine-tuned permissions for every pod, so it only has access to the resources it needs, not the whole cluster. Also, RBAC can specify fine-tuned permissions for developers, which protects a cluster from being taken over, in the event credential are stolen or compromised by a malicious user.

How Kubectl-RBAC Helps Simplify and Ultimately Strengthen RBAC Implementation

Kubectl-RBAC makes it easy to understand which permissions (not roles) a specific “User” has and compare it to what permissions a “User”/”Service Account” used in practice, by parsing the audit log. Before the tool, getting this information was a long and convoluted process. You had to:

1. Use the Kubectl command:
   kubectl describe rolebindings && kubectl describe clusterrolebindings
2. Search for the specified “User” & copy all the “Roles” assigned to them
3. Use the Kubectl command:
   kubectl describe roles && kubectl describe clusterrole
4. Search all the “Roles” from step-2 & copy the permissions
5. Get the audit log and parse it — note, this is not available in Kubectl, so we will show you how to do it later on GoogleCloud
6. Compare the information from Steps 4 & 5

To sum-up, it took too long and too many resources to get these insights. It’s mind numbing to think you need to do it for every existing and new “User.” We wrote this python extension, so we would never have to do it again — and now, with the Kubectl-RBAC tool, neither do you.

How Kubectl-RBAC Works

After you download and install the Kubectl-RBAC plugin, you can find all the “Roles” assigned to a specific “User” and all the permissions with a single Kubectl-RBAC command. The repository for the tool is located here.

1. First, you can check which “Users” you have with the command:

> kubectl describe clusterrolebinding | grep User
User user@octarinesec.com
User kubelet
User kube-apiserver
User kubelet
User system:node-problem-detector
User system:kube-controller-manager
User system:kube-scheduler
User system:kube-proxy

2. Next, if you want to inspect a specific “User,” e.g. user@octarinesec.com, you can enter the command:

> kubectl plugin rbac get-permissions user@octarinesec.com
[[{‘apiGroups’: [‘’],
‘resources’: [‘clusterrolebindings’,
‘clusterroles’,
‘roles’,
‘rolebindings’],
‘verbs’: [‘list’]
},
{‘apiGroups’: [‘’],
‘resources’: [‘clusterroles’, ‘clusterrolebindings’],
‘verbs’: [‘get’]
}]]

3. Now, if you want to compare what you found for the “User” with the permissions the “User” is really using, you can parse the audit log for the past seven days with the command:

> kubectl plugin rbac get-audited-permissions user@octarinesec.com kubectl_rbac/tests/audit_log.json
{…
‘rbac.authorization.k8s.io/v1/clusterroles/system:basic-user’: {‘io.k8s.authorization.rbac.v1.clusterroles.get’},
‘rbac.authorization.k8s.io/v1/clusterroles/system:certificates.k8s.io:certificatesigningrequests:nodeclient’: {‘io.k8s.authorization.rbac.v1.clusterroles.get’}, ‘rbac.authorization.k8s.io/v1/clusterroles/system:certificates.k8s.io:certificatesigningrequests:selfnodeclient’: {‘io.k8s.authorization.rbac.v1.clusterroles.get’},
‘rbac.authorization.k8s.io/v1/clusterroles/system:controller:attachdetach-controller’: {‘io.k8s.authorization.rbac.v1.clusterroles.get’},
‘rbac.authorization.k8s.io/v1/namespaces/default/rolebindings’: {‘io.k8s.authorization.rbac.v1.rolebindings.list’},
‘rbac.authorization.k8s.io/v1/namespaces/default/roles’: {‘io.k8s.authorization.rbac.v1.roles.list’}}

4. You can then check if there are any unused permissions (as seen in the log) granted to the specified “User,” with the command:

> kubectl plugin rbac get-unused-permissions user@octarinesec.com kubectl_rbac/tests/audit_log.json
{‘create’: set(),
‘delete’: set(),
‘get’: set(),
‘list’: set(),
‘patch’: set(),
‘update’: set(),
‘watch’: set()}

This will allow you to see if the “User” is configured properly, with the least privilege permissions.

5. If you find the “User” is not configured properly, you can use Kubectl-RBAC to configure the proper permissions, based on the audit-log, with:

> kubectl plugin rbac get-least-privilege user@octarinesec.com kubectl_rbac/tests/audit_log.json
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: octarine:user@octarinesec.com
namespace: default
rules:
- apiGroups:
- ‘’
resources:
- clusterroles
- clusterrolebindings
- rolebindings
- roles
verbs:
- list
- apiGroups:
- ‘’
resources:
- clusterroles
- clusterrolebindings
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: octarine:user@octarinesec.com
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: octarine:user@octarinesec.com
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: user@octarinesec.com

6. Lastly, you can pipe this output to roles.yaml and run kubectl -f create roles.yaml

Audit Log

If you don’t readily have the audit log, you will need to get it. The full documentation on the audit log can be found here. To enable and retrieve the audit log on GoogleCloud you must:

1. Enable all access logs:

cat <<EOF >> policy.json
“auditConfigs”: [
{
“service”: “allServices”,
“auditLogConfigs”: [
{ “logType”: “ADMIN_READ” },
{ “logType”: “DATA_READ” },
{ “logType”: “DATA_WRITE” },
]
},
]
EOF
gcloud projects set-iam-policy [PROJECT_ID] policy.json

2. Retrieve starting from “timestamp”:

gcloud logging read ‘resource.type=”k8s_cluster” AND timestamp>=”2018–05–17T14:50:00Z”’ — format=json > log.json

Summary

The open source tool, Kubectl-RBAC, allows you to quickly and easily establish least privilege for your Kubernetes clusters. When you use RBAC, you can improve the security of your applications and minimize the impact of any compromises within Kubernetes pods to ensure they attacks cannot spread to the entire cluster. This is the first of a number of measures we suggest you deploy to secure your cloud-native apps. For questions, you can direct message us on twitter@octarinesec and/or join the google group, kubectl-rbac. For more information on other best practices, please look for our blog series or contact us at info@octarinesec.com.