Kubernetes RBAC permissions model and how to add users to AWS EKS
So, you have decided to learn Kubernetes. The AWS flavor, more specifically, the AWS EKS service.
As you learn you think, Jesus effing Christ, why is this stuff so complex? And as you reach the topic of RBAC, it has all gotten much worse. You read some articles, including, of course, the article on AWS’ own documentation, and on the Kubernetes website. But there are still pieces missing.
But, fear not! This article aims to teach you to understand how RBAC works, specifically on AWS EKS. To fill in all the gaps from all the articles you have just read (or didn’t read). So you can add all the users you need, create and assign all the role bindings you need.
I won’t dive deep into all possible configurations. Instead, I will give you a complete instruction to add users and their privileges, with an emphasis in the bits that in my opinion were missing from all the articles I read when I had to learn RBAC for EKS, with which you will understand the logic and be able to create any permissions you need.
Prerequisites: to fully understand this content, you need to:
- Have a basic understanding of how Kubernetes works and be able to interact with it using kubectl. Know command such as kubectl apply -f , and kubectl get <object> -o yaml, and know what are Kubernetes namespaces.
- Understand the basic of AWS IAM: what are IAM users, how to create one, and their access keys.
- Have the AWS CLI and the access keys of your IAM users installed and configured.
- Have an AWS EKS cluster to play with. Preferably one you provisioned yourself, that you can safely play with and not worry about damaging it. Be able to configure connection to the cluster using the command aws eks update-kubeconfig.
It is out of scope of this article how to launch an EKS cluster. But let me just give you this tip. With this link you can use Terraform to quickly and reliably launch a basic EKS cluster. And to dispose of it when you are done, to stop incurring charges. Last time I checked an AWS EKS cluster costed around U$0.10/hour, plus the cost of the instances. Not a big deal, unless you forget it on.
How to add users to AWS EKS
If you have been reading some articles, you saw that Kubernetes does not manage users. It leaves that to the underlying platform. This means that, in AWS EKS, the users will come from IAM. You will be adding IAM users and IAM roles to your cluster.
Also if you have been reading the AWS article on adding users to EKS, you may have read that the cluster creator already has access to the cluster. It’s true; that’s how EKS is setup. The user that you used to created the cluster, or the role you were assuming when you created the cluster, that user/role is the cluster creator, and has admin access. You need that user/role to add more people to the cluster. If you lose access to that user/role, you can’t add any more users. Also notice the identification of the cluster creator for each cluster is not available in the EKS console. It does not tell you. You need to know or remember.
But an important takeaway in my opinion is: do not try to search for the cluster creator configuration inside your cluster. While you are learning, you may think you can just look up how the cluster creator is setup, so you can reproduce that configuration to the other users you want to add. But it’s just not there. It’s configuration is somehow hidden under the trunk, in the AWS-managed side. After all EKS is a managed service. Some parts of the system have been abstracted away from you for simplicity’s sake (or to minimize the crushing complexity of Kubernetes, is another way to put it). Just be aware not to waste your time searching for the cluster creator’s configuration.
In EKS, assigning users to the cluster starts with a configmap called aws-auth. All EKS clusters have it. Use this command to retrieve the source of the configmap and save it in a file called “aws-auth.yml”:
kubectl get configmap/aws-auth -n kube-system -o yaml > aws-auth.ymlIf you open it up, it should look like this.
apiVersion: v1
data:
mapRoles: |
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::465271127900:role/TestK8sCluster-nodegroup-role
username: system:node:{{EC2PrivateDNSName}}
kind: ConfigMap
metadata:
creationTimestamp: "2022-12-23T13:53:43Z"
name: aws-auth
namespace: kube-system
resourceVersion: "1006"
uid: 9f17c5f1-da1c-4f11-b6f5-66f3234c545cDon’t worry trying to understand all of this code just yet. What you need now is the block mapUsers, that goes under mapRoles. It looks like this:
mapUsers: |
- userarn: <the arn of your user>
username: <the username of your user, from IAM>Suppose you have a new IAM user called dumbuser. Adding him/her to the configmap, it would look like this:
apiVersion: v1
data:
mapRoles: |
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::465271127900:role/TestK8sCluster-nodegroup-role
username: system:node:{{EC2PrivateDNSName}}
mapUsers: |
- userarn: arn:aws:iam::465271127900:user/dumbuser
username: dumbuser
kind: ConfigMap
metadata:
creationTimestamp: "2022-12-23T13:53:43Z"
name: aws-auth
namespace: kube-system
resourceVersion: "1006"
uid: 9f17c5f1-da1c-4f11-b6f5-66f3234c545cNow apply this code to your cluster, with command:
kubectl apply -f aws-auth.ymlYou should get a confirmation that it was changed. Now let’s test it. For that, have a shell session with the access key for dumbuser setup in the AWS CLI.
If you are just switching from the cluster creator user you were using until now, remember you need to run update-kubeconfig again because this command writes not only your cluster address to file ~/.kube/config, but also your username.
Now that you have the access keys for dumbuser, try to retrieve the namespaces:
kubectl get nsYou should get result: Error from server (Forbidden): namespaces is forbidden: User “dumbuser” cannot list resource “namespaces” in API group “” at the cluster scope
If you tried to retrieve the namespaces with dumbuser before editting the configmap, you would get error: You must be logged in to the server (Unauthorized). See the difference. Unauthorized means the cluster has no idea who you are. Forbidden means you are a recognized user, but you do not have permissions.
So that part is done. We have added an IAM user to our AWS EKS cluster. Now let’s see:
How to add permissions to users in AWS EKS
Let’s see how dumbuser can get some permissions. Now we start talking about some objects you may have heard. First there is a cluster object called simply role. As you may infer, is used to grant someone, actions (verbs) on resources, for a specific namespace. See this example. This will allow someone (the one who has this role) access to get, watch and list pods, in the default namespace:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: default
name: pod-reader
rules:
- apiGroups: [""] # "" indicates the core API group
resources: ["pods"]
verbs: ["get", "watch", "list"]The cluster object named cluster role is the same, except that it is not restricted to a namespace. It applies to all namespaces of the cluster. And we are going to create this cluster role:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: getnamespaces
rules:
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list"]
apiVersion: rbac.authorization.k8s.io/v1This will allow someone to get and list namespaces. Exactly what we were trying before when dumbuser issued command kubectl get ns.
Save this last piece of code in a file and apply it to the cluster. The filename is unimportant. The cluster role will be created with the name specified in field metadata.name. I’m saving it as filename “getnamespaces.yml” therefore the command to apply it is:
kubectl apply -f getnamespaces.ymlYou will get a confirmation message. Just so you know, if you want to list all your roles and cluster roles:
kubectl get clusterrole
kubectl get role --all-namespacesNow you have a user… and a role. You just need to associate the two. Simple, right? In most systems this is a one-line command. Or some 3 or 4 mouse clicks, in a Windows program. Right? Right???
No. It can’t be that simple. After all, this is Kubernetes! To grant a role to a user, you need another entire YAML file with some 10 lines of code or more, apply this file to the cluster and the relationship between the user object and the role object is, itself, another object in the cluster.
These objects are called cluster role bindings, if they grant cluster roles to users, or just role bindings, if they grant roles to users. They don’t have an acronym so you need to type them whole in kubectl commands. Try this in your cluster:
kubectl get clusterrolebindings
kubectl get rolebindings --all-namespacesThese command will list you the cluster role bindings/role bindings you have and the roles they bind to. You will see that EKS comes with a lot of them preinstalled. To see the users they bind to, you will neet to get these objects individually, like these commands below.
kubectl get clusterrolebinding/<cluster role binding> -o yaml
kubectl get rolebinding/<role binding> -n <namespace> -o yamlOk, ok! Turns out there is a one-line command to create cluster role bindings and role bindings. But come on, man, don’t ruin the joke. It is kubectl create rolebinding <parameters> and kubectl create clusterrolebinding <parameters>. The effect is the same as using the yaml file. It will still create a whole other cluster object for what could have been a line in a user to role mapping table.
Let’s continue our exercise. We are trying to grant dumbuser the permission to view namespaces. We created a cluster role getnamespaces with the appropriate permissions. Now we need a cluster role binding. Like this:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: getnamespaces-crb
subjects:
- kind: User
name: dumbuser
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: getnamespaces
apiGroup: rbac.authorization.k8s.ioI have saved this file as crb.yml therefore the command to apply it is:
kubectl apply -f crb.ymlNow all is set. If you try again with dumbuser in your console, kubectl get ns, you should succeed, and see the preinstalled namespaces of a new EKS cluster:
NAME STATUS AGE
default Active 4h18m
kube-node-lease Active 4h18m
kube-public Active 4h18m
kube-system Active 4h18mNice! you added authentication and authorization to an IAM user on your EKS cluster. But we are not done, and it gets a bit more complex. But let’s see what we did so far. I have drawn you a diagram:
See how one cluster object relates to another cluster object. Except the user, who is not a cluster object, but instead a name, a reference from the platform (AWS IAM), that comes throught the aws-auth config map.
These names are pretty long. From now on let’s call the cluster role bindings CRBs, and the role bindings RBs, ok?
Groups: time to make it even more complex!
So far so good. You found out how to add a user to the cluster, and how to add permissions using roles/cluster roles and RBs/CRBs. And you know that role and cluster role are pretty much the same, except that one is namespace-specific and the other one is not.
But if you open up the aws-auth configmap in some existing cluster, you may find something like this:
data:
mapRoles: |
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::356198252393:role/eksctl-EKSTestDrive-nodegroup-ng-NodeInstanceRole-5C1P4COCU9W1
username: system:node:{{EC2PrivateDNSName}}
mapUsers: |
- userarn: arn:aws:iam::356198252393:user/eksadmin
username: eksadmin
groups:
- system:masters
kind: ConfigMap
metadata:
creationTimestamp: "2019-07-08T09:47:15Z"
name: aws-auth
namespace: kube-system
resourceVersion: "4169"
selfLink: /api/v1/namespaces/kube-system/configmaps/aws-auth
uid: 5df31bad-a165-11e9-90f3-12c84ad916b6In this aws-auth configmap example, user eksadmin is being added to the cluster. But through groups: system:masters. Just what is this? You won’t find a CRB or RB named system:masters. Because this is a group. These groups, unlike config maps, roles, cluster roles, RBs and CRBs, are not cluster objects; they are like the users: just a reference you find in the definition of objects.
Your AWS EKS cluster will come with this CRB called cluster-admin. Check it’s code:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
creationTimestamp: "2022-12-23T13:49:47Z"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: cluster-admin
resourceVersion: "145"
uid: 452ae590-d24a-4f3f-ad7c-d718e8a657bd
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:mastersThis CRB (cluster role binding), instead of binding to a user, is binding to a group. A group that is also seen in the aws-auth configmap. This is the connection. Let’s review:
- You add a user to aws-auth under groups: system:masters…
- Group system:masters is referenced in CRB cluster-admin;
- This CRB binds to a cluster role with the same name (cluster-admin) identified in section roleRef: of the CRB;
- Cluster role cluster-admin grants access to all verbs, all resources, all api groups.
Let’s see the code of cluster role cluster-admin, not to be confused with the cluster role binding of the same name:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
creationTimestamp: "2022-12-23T13:49:46Z"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: cluster-admin
resourceVersion: "83"
uid: 33d7ea82-e31b-44d5-b99f-2fef89604762
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- '*'
- nonResourceURLs:
- '*'
verbs:
- '*'This is how you retrieve the code of these two objects of the same name, just to be clear:
kubectl get clusterrolebinding/cluster-admin -o yaml > crb-cluster-admin.yml
kubectl get clusterrole/cluster-admin -o yaml > cr-cluster-admin.ymlLet’s put this group to work. Before, we added dumbuser to aws-auth configmap, and then created a cluster role binding between dumbuser and a cluster role. Let’s undo that and make use of this group instead.
Drop the objects we created before:
kubectl delete clusterrole/getnamespaces
kubectl delete clusterrolebinding/getnamespaces-crbAnd now add dumbuser in the aws-auth configmap, but now with this group, system:masters.
apiVersion: v1
data:
mapRoles: |
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::465271127900:role/TestK8sCluster-nodegroup-role
username: system:node:{{EC2PrivateDNSName}}
mapUsers: |
- userarn: arn:aws:iam::465271127900:user/dumbuser
username: dumbuser
groups:
- system:masters
kind: ConfigMap
metadata:
creationTimestamp: "2022-12-23T13:53:43Z"
name: aws-auth
namespace: kube-system
resourceVersion: "33289"
uid: 9f17c5f1-da1c-4f11-b6f5-66f3234c545cNow give it a test. Go back to your shell session with dumbuser configured. dumbuser Is now a cluster admin and can do other stuff, such as:
kubectl get all --all-namespacesOur diagram now looks like this.
So, before it was: users > rolebindings > roles > permissions. Now there’s one more layer: users > groups > rolebindings > roles > permissions.
This should have given you the understanding of:
- How you are meant to add IAM users/roles to AWS EKS clusters;
- What is the cluster creator and to be aware that he is the only user with access (initially) and to never try to search for it inside the cluster;
- How objects roles, cluster roles, role bindings, cluster role bindings can be used to grant permissions (verbs, resources) to users;
- How to use groups from aws-auth which represent one more layer between your users and their permissions.
How to add IAM roles to AWS EKS
Now that we have seen this all, I can show you how to add IAM roles.
Here is the entry you need to add under mapRoles:
- rolearn: <ARN of your role>
username: <some username you need to choose>
groups:
- <a Kubernetes group (optional)>See how it looks in real life, supposing you want to add a role called somerole, and for whoever IAM user that assumes this role, you want them to get the privileges granted by the system:masters Kubernetes group.
apiVersion: v1
data:
mapRoles: |
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::465271127900:role/TestK8sCluster-nodegroup-role
username: system:node:{{EC2PrivateDNSName}}
- rolearn: arn:aws:iam::465271127900:role/somerole
username: username1
groups:
- system:masters
kind: ConfigMap
metadata:
creationTimestamp: "2022-12-25T23:24:38Z"
name: aws-auth
namespace: kube-system
resourceVersion: "3737"
uid: 37d2afca-f283-4245-8fa3-c2b69f1fdd10Notice that you need to choose a username. In this case, I used username1. Then, you need to create CRB or RB objects to this username. Or, as I did, you grant username1 to group system:masters directly in the config map, so you don’t need any more CRB/RB objects.
If you were reading some articles and you are still using the approach of looking at existing config and replicating it (I won’t blame you, I do that a lot), you may have been led to think that field username can have value system:node:{{EC2PrivateDNSName}}. Not really. It does not work if you use that. You need a regular username there.
Another important note, if your role ARN includes a path, like this:
arn:aws:iam::111122223333:role/my-team/developers/role-nameYou need to remove the path. It must look like this:
arn:aws:iam::111122223333:role/role-nameMoreover, to test access to the cluster from the IAM role, there’s a lot involved. You need a role. This role needs permission eks:describeCluster on this cluster. This role needs to have a trust policy allowing this user to assume this role. This user needs permission sts:assumeRole to that role. Then you go to the shell with the user, issue command aws sts assume-role, and reconfigure your credentials to the cluster (aws eks update-kubeconfig). All of this can be a little laborious and is off topic, I’m afraid. This article is already way too long.
Just notice what you don’t need: you don’t need to grant IAM permissions on the cluster to that role (somerole). Because all permissions are granted through aws-auth config map or through CRB/RB objects. In IAM, the role in question just needs permission eks:describeCluster so it can execute aws eks update-kubeconfig.
Bonus content: searching within cluster objects
Let’s say there’s a user. And in the aws-auth configmap, he is given a certain group. But you don’t know which rolebindings/clusterrolebindings are connecting this group to which roles. How do you search for this? Extracting each role binding/cluster role binding to filter their contents for the group name takes too long. But since I discovered this kind of Bash script, I use it a lot. I’m not going to turn this into a Bash programming article. But, if you are in to it, check how this script lists all cluster role bindings, then extract their source code, then filters it by the group name, informing you of all CRBs that match string system:masters.
¬#!/bin/bash
for ITEM in $(kubectl get clusterrolebinding | grep -v ^NAME | awk '{print $1}'); do
SIZ=$(kubectl get clusterrolebinding/$ITEM -o yaml | grep "system:masters" | wc -l);
if [[ SIZ -ge 1 ]]; then
echo "found matching clusterrolebinding: $ITEM"
else
echo "not here..."
fi
doneBefore you go, let me know in the comments. What do you think Kubernetes was created for?
a) To host and manage containers.
b) To separate men from boys.
c) To make grown men cry.
d) So that platform engineers never feel too cocky.
This code was tested on:
- AWS EKS: server version v1.23.14-eks-ffeb93d (latest as of now)
- kubectl: version v1.23.7-eks-4721010
- AWS CLI 2.7.18
- Operating system below kubectl and AWS CLI: Ubuntu 20.04