Running Gitlab runner behind a proxy and with a private container registry

Source: https://www.onyxpoint.com/account-level-ci-access-management-with-gitlab-setuid-runners/

Setting up Gitlab runner to work properly behind an Enterprise internet proxy and using Docker-in-Docker for your CI builds with a private container registry is challenging. Here are the steps you need to follow to set this up on an Ubuntu VM.

Step 1: Install Docker on your fresh Ubuntu VM
We’ll export our Enterprise internet proxy into the http_proxy and https_proxy environment variables, and skip proxy for any internal domains we don’t want to access via the proxy into the no_proxy variable. Then we’ll install Docker. Replace proxyIP and proxyPort with your enterprise proxy IP and port.

ubuntu@gitlabrunner:~$ export http_proxy=http://proxyIP:proxyPort/ https_proxy=http://proxyIP:proxyPort/ no_proxy=localhost,127.0.0.1,gitlab.internaldomain.com
$ sudo apt update
$ sudo apt upgrade -y
$ curl -fsSL https://get.docker.com -o get-docker.sh
$ sh get-docker.sh
$ sudo usermod -aG docker ubuntu
$ exit # SSH back into the VM after exit

Step 2: Add your proxy to Docker engine

$ sudo mkdir -p /etc/systemd/system/docker.service.d
$ sudo vi /etc/systemd/system/docker.service.d/http-proxy.conf
# Add the text below to the http-proxy.conf file
[Service]
Environment="HTTP_PROXY=http://proxyIP:proxyPort/" "HTTPS_PROXY=http://proxyIP:proxyPort/" "NO_PROXY=localhost,127.0.0.1,gitlab.internaldomain.com"
$ sudo systemctl daemon-reload
$ sudo systemctl restart docker
$ docker run hello-world # If this doesn't work there is something wrong in your docker config

Step 3: Setup insecure registries on Docker
Behind enterprise, we run our gitlab container registry with self-signed certificates. In order to allow Docker to access these registries, we need to setup the insecure registry inside Docker.

$ sudo vi /etc/docker/daemon.json
# Add the text below to the daemon.json file
{
  "insecure-registries" : ["gitlab.internaldomain.com:4567"]
}
$ sudo systemctl restart docker
ubuntu@gitlabrunner:~$ docker login gitlab.internaldomain.com:4567
Username: ci_user
Password:
WARNING! Your password will be stored unencrypted in /home/ubuntu/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
$ sudo cat /home/ubuntu/.docker/config.json
{
 "auths": {
  "gitlab.internaldomain.com:4567": {
   "auth": "amXua7luczpidFExZHUmZXI="
  }
 },
 "HttpHeaders": {
  "User-Agent": "Docker-Client/19.03.9 (linux)"
 }

Step 4: Run Gitlab runner using docker

$ docker run -d --name gitlab-runner --restart always \
  -v /srv/gitlab-runner/config:/etc/gitlab-runner \
  -v /var/run/docker.sock:/var/run/docker.sock \
  gitlab/gitlab-runner:v12.10.2
$ docker ps
CONTAINER ID        IMAGE                           COMMAND                  CREATED             STATUS              PORTS               NAMES
10546fd70048        gitlab/gitlab-runner:v12.10.2   "/usr/bin/dumb-init …"   9 seconds ago       Up 6 seconds                            gitlab-runner

This will keep all gitlab-runner config in your Ubuntu VM directory /srv/gitlab-runner/config

Step 5: Make gitlab runner trust your self-signed certificates
Download the self-signed certificate from your gitlab and place this into the /srv/gitlab-runner/config directory.

ubuntu@gitlabrunner:~$ openssl s_client -showcerts -connect repo.tejdrive.com:12443 </dev/null 2>/dev/null|openssl x509 -outform PEM >gitlab.internaldomain.com.pem
$ sudo mkdir -p /srv/gitlab-runner/config/certs
$ sudo cp gitlab.internaldomain.com.pem /srv/gitlab-runner/config/certs/gitlab.internaldomain.com.crt

Step 6: Register the runner in Gitlab
Log into the gitlab-runner container and run the registration command. You can get the gitlab-ci registration token from your Gitlab UI under Settings > CICD > Runners.

$ docker exec -it 10546fd70048 bash # 10546fd70048 is containerID
root@10546fd70048:/# gitlab-runner register
Runtime platform                                    arch=amd64 os=linux pid=76 revision=c5874a4b version=12.10.2
Running in system-mode.
Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
https://gitlab.internaldomain.com:12443/
Please enter the gitlab-ci token for this runner:
MC5QPHzW554C4YZAwWq8
Please enter the gitlab-ci description for this runner:
[10546fd70048]: gitlabrunner01
Please enter the gitlab-ci tags for this runner (comma separated):
Registering runner... succeeded                     runner=MC5QPHzW
Please enter the executor: parallels, shell, docker-ssh, ssh, virtualbox, docker+machine, docker-ssh+machine, kubernetes, custom, docker:
docker
Please enter the default Docker image (e.g. ruby:2.6):
alpine:latest
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

Step 7: Edit gitlab runner’s config and add docker registry authentication

ubuntu@gitlabrunner01:~$ docker exec -it 10546fd70048 bash
root@10546fd70048:/# vi /etc/gitlab-runner/config.toml
concurrent = 8
check_interval = 0
[session_server]
  session_timeout = 1800
[[runners]]
  name = "gitlabrunner01"
  url = "https://gitlab.internaldomain.com:12443/"
  token = "XkBAtmzSvexwxQPVEeh-"
  executor = "docker"
  environment = ["DOCKER_AUTH_CONFIG={\"auths\":{\"gitlab.internaldomain.com:4567\":{\"auth\":\"amVua2luczpidSExZHUmZXI=\"}}}", "https_proxy=http://proxyIP:proxyPort/", "http_proxy=http://proxyIP:proxyPort/", "HTTPS_PROXY=http://proxyIP:proxyPort/", "HTTP_PROXY=http://proxyIP:proxyPort/", "no_proxy=localhost,127.0.0.1,gitlab.internaldomain.com", "NO_PROXY=localhost,127.0.0.1,gitlab.internaldomain.com"] # Add
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.docker]
    tls_verify = false
    image = "alpine:latest"
    privileged = true # Update
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/certs/client", "/cache"] # Update
    dns = ["10.XYZ.ABC.EFG", "10.XYZ.ABC.EFG", "8.8.8.8"] # Update
    shm_size = 0
root@10546fd70048:/# gitlab-runner restart
root@10546fd70048:/# exit

Make sure to update the proxyIP:proxyPort and the internal DNS server IPs above.

That’s it, your Ubuntu VM should be able to run your internal Gitlab’s CI builds while accessing container images from your own private registry.