Deploying GitHub Action Runners on GKE with dind-rootless

TLDR: This article describes the steps to configure and deploy self-hosted GitHub Action Runners using docker:dind-rootless to Google Kubernetes Engine (GKE). If you’re already familiar with the concepts and would like to jump into the guide, please skip to the “Installing GitHub ARC with Helm on GKE” section below. For a more in-depth exploration of dind-rootless on GKE, please see the next article “Troubleshooting GitHub Action Runners with dind-rootless on GKE: Deep Dive”.

Background

Software organizations that center their development processes on GitHub will often use GitHub Actions for CI/CD. GitHub Action Workflows typically run on GitHub’s own infrastructure via GitHub’s Action Runners. However, organizations have the option to host their own GitHub Action Runners in an environment of their choosing. This option is results in what are typically described as self-hosted Action Runners.

The GitHub documentation for hosting your own runners describes some common motivation vectors for organizations that choose to self-host:

Self-hosted runners offer more control of hardware, operating system, and software tools than GitHub-hosted runners provide. With self-hosted runners, you can create custom hardware configurations that meet your needs with processing power or memory to run larger jobs, install software available on your local network, and choose an operating system not offered by GitHub-hosted runners. Self-hosted runners can be physical, virtual, in a container, on-premises, or in a cloud.

About GitHub Action Runners

GitHub Action Runners consist of two primary components: Action Runner Controller and Runner Scale Set. The Action Runner Controller (ARC) manages the life cycle of the Runner Scale Set, which in turn creates the Pods ultimately responsible for running the Action Runner Workflows, thus fulfilling the CI/CD requirements of self-hosted GitHub Actions runners on the environment of an organization’s choice.

GitHub Action Runners may be deployed to Google Kubernetes Engine (GKE) using two GitHub supported Helm charts: the Action Runner Controller Helm chart and the Runner Scale Set Helm chart. The ARC Helm chart configuration we will use is essentially turn-key, so we will focus the Runner Scale Set configuration.

Configuring the Runner Scale Set Helm chart

The Runner Scale Set Helm supports 2 first-class (named) modes, with a 3rd option for a custom mode (which we will use for dind-rootless)

1. Kubernetes Mode
2. Dind (Docker-in-Docker) Mode
3. Custom Mode

Kubernetes mode configures ARC to use container hooks to create a new pod in the same namespace to run the Action Workflow. Kubernetes mode is comparably less resource intensive than Dind mode and works well for Action Workflows that are not required to build Docker containers. It is important to note that Kubernetes Mode may be used to build containers when using daemon-less tools, such as Kaniko, that can build from a Dockerfile.

Dind mode is required for Action Workflows that depend on the Docker daemon to build containers. Dind (Docker-in-Docker) mode ensures that the Runner container within the Runner Pod has access to a Docker daemon, most commonly exposed through a Unix socket. If your Action Workflows depends on the Docker CLI, or other features specific to Docker, you will likely be required to run the Runner Scale Set in Dind mode, or a custom mode with a Docker daemon.

Dind Mode works well out of the box on GKE, but requires the Docker daemon to run as root within the container in the pod. In the interest of following the security principal of least privilege, we want to minimize our attack service by running pods as non-root users. Sometimes there may be no viable alternative to running your container as root, especially for a container with comparably low-level resource requirements like the docker:dind container. One can argue about the trade-offs of defense-in-depth security postures such as this (shouldn’t the container runtime properly isolate the containers from the host system?!) but when we have a viable non-root alternative that meets our feature requirements we should likely take it.

Custom Mode allows the arbitrary configuration of a Pod this is used as the runtime for Action Runner Workflows. Any other Action Runner Scale Set configuration then Kubernetes or Dind mode is required to use this opion. The dind-rootlesscontainer is configured in the Runner Scale Set Helm chart via a custom template configuration in the Helm values.yaml (Note: the sample configuration needs to be modified to run on GKE, shown in he guide below). The dind-rootless container uses Rootlesskit to create a fake root environment capable of accessing most of the resources that the standard dindcontainer requires. Rootlesskit was built specifically for Docker and Kubernetes and leverages Linux Namespaces, along with some complex uid/gid management, to meet it’s requirements. For a detailed explanation for customizing dind:rootless for GKE, please see the next article titled “Troubleshooting GitHub Action Runners with dind-rootless on GKE: Deep Dive”

Configuring the GKE Cluster

GKE clusters are highly configurable. There are layers of configuration options for networking, security, availability, and other areas. The GKE Cluster we will configure will use a minimum of configuration options, but it is recommended that organizations self-hosting GitHub Action Runners to configure their clusters using best security practices for CI/CD systems.

The GKE Cluster operation mode, Autopilot or Standard, is one of the highest level components of GKE configuration. Although Autopilot is the preferred operation mode for most GKE workloads, running dind-rootless forces the cluster to run in Standard operation mode. The reason is that thedind-rootless container is required to run in privileged mode (i.e. containers.securityContext.privileged = true).

In the interest of conserving resources, this guide creates a Public Zonal GKE Cluster. In production, it is recommended to create a Private Regional GKE cluster for higher security and availability. The Private GKE Cluster must be able to establish egress connections with the required GitHub APIs.

Now that we have a knowledge base for how we intend to configure the self-hosted Action Runner, let’s put is to use and walk through the installation process.

Installing GitHub ARC with Helm on GKE

The following instructions serve as a sample guide to installing and running Self-Hosted Action Runner Controllers with dind-rootless on GKE.

Pre-Requirements

1. An active Google Cloud account and project.
2. Access to the Cloud Shell in the Google Cloud Console, or an environment with gcloud, kubectl, and helm.
3. A GitHub Repository configured with a GitHub Actions Workflow that uses a Docker Daemon to build a container image.
4. A GitHub Access Token with access to your Repository. A Personal Access Token (classic) is a fairly straightforward way to validate your Action Runner integration.

Install

1. Navigate to your project in the Google Cloud Console and open the Cloud Shell
2. Enable required APIs

gcloud services enable container.googleapis.com

3. Create a GKE Cluster.

Note: You may use any GKE Cluster in Standard Operation mode that has connectivity to the required GitHub APIs.

gcloud container clusters create action-runner-cluster \
    --zone us-central1-a \
    --node-locations us-central1-a

3. Set thekubectl context to the new cluster

gcloud container clusters get-credentials action-runner-cluster

4. Install the Action Runner Controller via Helm Chart

NAMESPACE="arc-systems"
helm install arc \
    --namespace "${NAMESPACE}" \
    --create-namespace \
    oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller

5. Configure the Runner Scale Set Helm Chart Values

The following document should be written to a file named values.yml located in the same directory where you will run the helm install command for the Runner Scale Set Helm chart. For all options in values.yaml see the action-runners-controller repo.

Note: you will need to change the githubConfigUrl and githubConfigSecret to match your environment:

Note 2: the values.yaml below diverges slightly from the sample GitHub values.yaml so that it can run on GKE. In this version, the Docker Daemon socket runs on `unix:///home/runner/var/run/docker.sock` as opposed to the default `unix:///var/run/docker.sock`. For a detailed explanation for this change, please see the next article titled “Troubleshooting GitHub Action Runners with dind-rootless on GKE: Deep Dive.”

## githubConfigUrl is the GitHub url for where you want to configure runners
## ex: https://github.com/myorg/myrepo or https://github.com/myorg
## githubConfigUrl: "https://github.com/[user]/[repo]"
## Note: specified on command line outside of values.yml

## githubConfigSecret is the k8s secrets to use when auth with GitHub API.
## You can choose to use GitHub App or a PAT token
## githubConfigSecret: "my-token"
## Note: specified on command line outside of values.yml

## maxRunners is the max number of runners the autoscaling runner set will scale up to.
maxRunners: 5

## minRunners is the min number of idle runners. The target number of runners created will be
## calculated as a sum of minRunners and the number of jobs assigned to the scale set.
minRunners: 0

#runnerGroup: "my-custom-runner-group"

## name of the runner scale set to create.  Defaults to the helm release name
runnerScaleSetName: "arc-runner-set"

## template is the PodSpec for each runner Pod
## For reference: https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec
template:
  spec:
    initContainers:
    - name: init-dind-externals
      image: ghcr.io/actions/actions-runner:latest
      command: ["cp", "-r", "-v", "/home/runner/externals/.", "/home/runner/tmpDir/"]
      volumeMounts:
        - name: dind-externals
          mountPath: /home/runner/tmpDir
    - name: init-dind-rootless
      image: docker:dind-rootless
      command:
        - sh
        - -c
        - |
          set -x
          cp -a /etc/. /dind-etc/
          echo 'runner:x:1001:1001:runner:/home/runner:/bin/ash' >> /dind-etc/passwd
          echo 'runner:x:1001:' >> /dind-etc/group
          echo 'runner:100000:65536' >> /dind-etc/subgid
          echo 'runner:100000:65536' >>  /dind-etc/subuid
          chmod 755 /dind-etc;
          chmod u=rwx,g=rx+s,o=rx /dind-home
          chown 1001:1001 /dind-home
      securityContext:
        runAsUser: 0
      volumeMounts:
        - mountPath: /dind-etc
          name: dind-etc
        - mountPath: /dind-home
          name: dind-home
    containers:
    - name: runner
      image: ghcr.io/actions/actions-runner:latest
      command: ["/home/runner/run.sh"]
      env:
        - name: DOCKER_HOST
          value: unix:///home/runner/var/run/docker.sock
      securityContext:
        privileged: true
        runAsUser: 1001
        runAsGroup: 1001
      volumeMounts:
        - name: work
          mountPath: /home/runner/_work
        - name: dind-sock
          mountPath: /home/runner/var/run
    - name: dind
      image: docker:dind-rootless
      args: ["dockerd", "--host=unix:///home/runner/var/run/docker.sock"]
      securityContext:
        privileged: true
        runAsUser: 1001
        runAsGroup: 1001
      volumeMounts:
        - name: work
          mountPath: /home/runner/_work
        - name: dind-sock
          mountPath: /home/runner/var/run
        - name: dind-externals
          mountPath: /home/runner/externals
        - name: dind-etc
          mountPath: /etc
        - name: dind-home
          mountPath: /home/runner
    volumes:
    - name: work
      emptyDir: {}
    - name: dind-externals
      emptyDir: {}
    - name: dind-sock
      emptyDir: {}
    - name: dind-etc
      emptyDir: {}
    - name: dind-home
      emptyDir: {}

6. Install the Runner Scale Set Helm chart

INSTALLATION_NAME="arc-runner-set"
NAMESPACE="arc-runners"
GITHUB_CONFIG_URL="https://github.com/[user]/[repo]"
GITHUB_PAT="[my-access-token]"
helm upgrade "${INSTALLATION_NAME}" \
    --namespace "${NAMESPACE}" \
    --create-namespace \
    --set githubConfigUrl="${GITHUB_CONFIG_URL}" \
    --set githubConfigSecret.github_token="${GITHUB_PAT}" \
    oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set

7. Trigger you GitHub Actions Workflow that builds a Docker images and look for success in the GitHub Action Logs!

Conclusion

Building Docker images on GKE using self-hosted Action Runners is entirely possible using the dind-rootless configuration. For a more detailed analysis of the change to the dind-rootless template in this article, please see the next article titled “Troubleshooting GitHub Action Runners with dind-rootless on GKE: Deep Dive”

Did I miss some important details, or is there anything else that I can help clarify? Please let me know in the comments. Thanks for your time!

Resources

1. Action Runner Controller Helm chart
2. Runner Scale Set Helm chart
3. Rootless Kit