5 min readMar 24, 2020

I just finished installing a GitLab runner on Kubernetes (on a namespace) and it wasn't as straightforward as I though it will be. So I decided to share my experience and hopefully easy the process for you.

First a disclaimer, I'm using GitLab 10.8 (current version is 12.9) but the process should be the same.

The goal here is to be able to run GitLab CI/CD, for that we need to run a GitLab runner somewhere and register it to our GitLab project. There is also the possibility of having shared runners but that won't be described here, although I believe it's pretty much the same setup.

We want to use the Kubernetes executor in the GitLab Runner so that each job in the GitLab pipeline is executed as a pod in our kubernetes cluster.

So we need to install the runner somewhere and we want that somewhere to be our kubernetes cluster. I assume that you already have a functioning kubernetes cluster and you can perform kubectl command on it.

So the prerequisites:

Functioning Kubernetes cluster where you have admin privileges
kubectl installed and configured to connect with the above cluster
helm installed (version 3)

We will follow the following step (detailed later)

Create a kubernetes namespace gitlab-runner
Create a role under gitlab-runner namespace with the rules that allow to create, delete, exec and access the logs of pods.
Assign / bind the role to the service account system:serviceaccount:gitlabrunner:default, which is the default service account for that namespace and that has no permissions by default
Use helm to install gitlab runner on kubernetes on the gitlab-runner namespace

Create a k8s namespace

I do prefer to have all the pods related to GitLab runner on a separate namespace gitlab-runner. The procedure to create a new namespace is described in the kubernetes documentation.

Create a file gitlab-runner-namespace.yaml that describes the new namespace:

and create the namespace by running kubectl create -f gitlab-runner-namespace.json.

Now check that the gitlab-runner namespace its there:

# kubectl get namespace
NAME                   STATUS   AGE
default                Active   73d
gitlab-runner          Active   15h
...

Create a role

Remember, we are going to install GitLab runner configured to use Kubernetes executor. That means that this GitLab runner will create a pod in the kubernetes cluster namespace for each job in each pipeline. So in order to do this it needs access to the Kubernetes API.

When the gitlab runner is installed in this namespace is going to use the service account system:serviceaccount:gitlab-runner:default (that is a service account under gitlab-runner namespace). That service account has no permission whatsoever so the gitlab runner won’t be able to create any pod.

We need to create a role with proper permissions and assign this role to the system:serviceaccount:gitlab-runner:default service account.

First create a file gitlab-runner-gitlab-runner-role.ymldescribing the role:

gitlab-runner-gitlab-runner-role.yml

This role allows to create and delete pods, to execute them and also access their logs. I couldn’t find a list of required permissions in the GitLab Runner documentation but I checked empirically and those seem to be the only ones used by the runner.

Then runkubectl create -f gitlab-runner-gitlab-runner-role.yaml
(or kubectl replace if you already have that role and want to overwrite) .

Now verify that the role it’s there under gitlab-runner namespace

# kubectl get --namespace=gitlab-runner role
NAME            AGE
gitlab-runner   4h7m# kubectl describe role --namespace gitlab-runner gitlab-runner
Name:         gitlab-runner
Labels:       <none>
Annotations:  <none>
PolicyRule:
  Resources  Non-Resource URLs  Resource Names  Verbs
  ---------  -----------------  --------------  -----
  pods/exec  []                 []              [create]
  pods/log   []                 []              [get]
  pods       []                 []              [list get watch create delete]

Assign the role to the service account

In the previous step we have created the role but we still need to bind this role to the service account that the gitlab runner will use.

# kubectl create rolebinding --namespace=gitlab-runner gitlab-runner-binding --role=gitlab-runner --serviceaccount=gitlab-runner:default

Then verify that the rolebinding is actually there with:

# kubectl get --namespace gitlab-runner rolebinding
NAME                    AGE
gitlab-runner-binding   4h16m

Install gitlab runner on Kubernetes using Helm

Ok, now we are ready to actually install GitLab runner on Kubernetes. Here the process is straighforward and it’s described in the documentation.

Basically we only need 2 configuration parameters: the url of the gitlab installation and a token. The gitlab runner will register itself against the gitlab installation (given by the URL) using the token (which determines which project the runner is assigned to).

You can get both the URL and the token by navigating to your project in GitLab , then Settings (at the bottom of the left sidebar) and CI/CD. You will find the URL and token under the section “ Setup a specific Runner manually”.(e.g url https://gitlab.mycompany.com and token a8d8f8df3fsfa)

Now we need to past that url and token into a file called values.yml :

gitlabUrl: https://gitlab.mycompany.com/
runnerRegistrationToken: "ssssssssssss"

There is other configuration parameters that you can add there but those two are the bare minimum.

Now the actual gitlab runner installation goes like this:

# helm repo add gitlab https://charts.gitlab.io
# helm install --namespace gitlab-runner gitlab-runner -f values.yaml gitlab/gitlab-runner

This will start up a pod with gitlab runner, the gitlab runner will autoregister with the gitlab instance / token specified in values.yaml and if you should see it in you GitLab project under Settings > CI/CD > Runners

You can also verify that the GitLab runner is shown as a pod in the kubernetes cluster with:

kubectl get --namespace gitlab-runner pod
 NAME                                           READY   STATUS    RESTARTS   AGE
gitlab-runner-gitlab-runner-xxxxx-yyyy          1/1     Running   0          19h

Testing that it works

Now it’s just a matter of adding a .gitlab-ci.yml file to your project:

job1:
  script: echo "hi there"

Then the pipeline would run and you can see in the output for the job that it was executed on kubernetes

If something fails you can start the troubleshooting by accessing the logs of the gitlab runner in kubernetes with the following command:

# kubectl logs --namespace gitlabrunner gitlab-runner-gitlab-runner-xxxxxxxxx
ERROR: Job failed (system failure): pods is forbidden: User "system:serviceaccount:gitlabrunner:default" cannot create resource "pods" in API group "" in the namespace "gitlab-runner"  duration=42.095493ms job=37482 project=yyy runner=xxxxxxx

For example in the above example you can see that the the service account is missing the permissions to create pods (this example was taken before the role was assigned to the service account).

Conclusion

Setting up GitLab runners straightforward for the most part. The only hurdle comes from using kubernetes namespaces and figuring out what set of permissions does need to be assigned to the service account so that the GitLab runner is able to create, delete, execute and access logs of pods.