K8s : Creating a Dedicated User (ServiceAccount) With Restricted Permissions for GitLab

Is it right to give GitLab admin permission to run kubectl commands in pipelines? Imagine that GitLab has been hacked, with admin permissions, they will be able to do whatever they want within the cluster and cause huge damage. Therefore, for security best practice, it is necessary to restrict the permissions given to Gitlab and grant as many permissions as necessary. Cluster wide permission should not be given to perform limited tasks in a certain namespace.

So how should we grant these permissions? By creating a role and connecting it to the service account with rolebinding? So how do we attach this serviceaccount to GitLab? Is it necessary to use a token?

First of all, we need a kubeconfig file for the kubectl commands to work. As we know, kubeconfig is the default way to authenticate to a Kubernetes cluster. We need to restrict the admin privileges in the config file in the .kube folder in our cluster by preparing another config file. Later, we are gonna give that kubeconfig file that we will generate for the ServiceAccount to GitLab.

We first create a namespace named myns with the following command:

kubectl create ns myns

Then we create a file named myrole.yaml and write the following content:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: myns
  name: myrole
rules:
- apiGroups: [""] # indicates the core API group
  resources: ["pods", "services", "secrets"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["extensions", "apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

With the following command, we ensure that the role named myrole is created in the myns namespace. With this role, we will give the owner of the role the permission to perform get, list, watch, create, update, patch and delete operations for pod, service, secret and deployment objects.

kubectl apply -f myrole.yaml

Now it’s time to create a service account named mysa:

kubectl create serviceaccount mysa --namespace=myns

In order to bind the role and the ServiceAccount together, it is now necessary to create a Rolebinding named myrb:

kubectl create rolebinding myrb \
  --role=myrole \
  --serviceaccount=myns:mysa \
  --namespace=myns

What we need now is the authentication token for the service account. To do this, we first create a secret called mysa-token:

apiVersion: v1
kind: Secret
metadata:
  name: mysa-token
  namespace: myns
  annotations:
    kubernetes.io/service-account.name: mysa
type: kubernetes.io/service-account-token

kubectl apply -f mysa-token.yaml

With the following command, we get an output where we can display the token as follows:

kubectl describe secrets mysa-token -n myns

Name:         mysa-token
Namespace:    myns
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: mysa
              kubernetes.io/service-account.uid: a3d2579f-483a-4547-8ae9-a1797577fe34

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1107 bytes
namespace:  4 bytes
token:      eyJhbGciOiJSUzXh________fT9qb4Zw

To get the value of the service account authentication token in a MacOS, Linux, or Unix environment:

TOKEN=`kubectl -n myns get secret mysa-token -o jsonpath='{.data.token}' | base64 --decode`

The following command adds the token of the ServiceAccount named mysa at the bottom of the $HOME/.kube/config file.

kubectl config set-credentials mysa --token=$TOKEN

Now, we can create a file called gitlab-config.yaml, copy and paste the content of the config file as it is, and return the config file to its previous state and maintain admin privileges on the cluster. The content of the new config file that we will give to GitLab, named gitlab-config.yaml:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTi_____DQVRFLS0tLS0K
    server: https://172.31.44.11:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: mysa
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: LS0tLSV______tLS0tCg==
    client-key-data: LS0tLS_____URSBLRV0tLQo=
- name: mysa
  user:
    token: eyJhbGciOiJSUz_______Xaq8jX-nMfT9qb4Zw

(Don’t forget to change the user section in the context section.)

It’s time to define the KUBECONFIG variable. If the KUBECONFIG environment variable doesn’t exist, kubectl uses the default kubeconfig file, $HOME/.kube/config. If the KUBECONFIG environment variable does exist, kubectl uses an effective configuration that is the result of merging the files listed in the KUBECONFIG environment variable. (1)

After using the export KUBECONFIG=gitlab-config.yaml command on the master node, try the kubectl get po command.

If your output is like that:

“Error from server (Forbidden): pods is forbidden: User “system:serviceaccount:myns:mysa” cannot list resource “pods” in API If you get the output of group “” in the namespace “default”

and

if you have no problems with the kubectl get po -n myns command, you are now ready to send the GitLab kubeconfig file by creating a variable named KUBE_CONFIG and putting the content of the config file as value:

Let’s ask the GitLab to create a secret object in a namespace called my-micro-service and see that the pipeline fails. Thus, we have verified that it cannot create the relevant objects in a namespace other than myns namespace. This is the security best practice…

Notes:

(1) https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/