GitLab + Runner + Registry + CI/CD Stack Setup and Management Best Practices by GitOps #1

Published in

Trendyol Tech

6 min readJan 10, 2020

With this article series, I wanna tell end-to-end experience and best practices, while our infrastructure transformation at Trendyol Tech. The company is the highest traffic and volumed e-commerce website in the country and region. The technology department consists of 20+ teams with 400+ members. Regarding to complexity concerns, we started to manage our resources & applications by using GitLab with GitOps methodologies, and easily track who/when/why did changes.

“GitOps: versioned CI/CD on top of declarative infrastructure. Stop scripting and start shipping.”
— Kelsey Hightower

Simplicity is our main principle. Due to this vision/goal, we selected GitLab and achieved a complete CI/CD toolchain and VCS. Let’s start, how we installed/managed those pieces step-by-step. 💪

Step 1: Provisioning Machine and Requirements

We are provisioning the node (8 Core, 16GB Memory Ubuntu 16.04.6 OS), by installing Docker Engine CE and initialize Swarm Mode for providing a lightweight container orchestration to serve GitLab.

#Install Docker Engine CE
~$ curl -fsSL https://get.docker.com/ | sh#Check Version
~$ docker -v
Docker version 19.03.5, build 633a0ea838#Init Docker Swarm Mode
~$ docker swarm init --advertise-addr 192.168.10.10#Check Nodes
~# docker node ls
ID        HOSTNAME STATUS AVAILABILITY MANAGER-STATUS ENGINE-VERSION
uz73dfr * node1    Ready  Active       Leader         19.03.5#Create Network
~$ docker network create --driver=overlay --attachable ty#Check Network
~# docker network ls
NETWORK-ID NAME            DRIVER  SCOPE
0b6cdf1    bridge          bridge  local
1306494    docker_gwbridge bridge  local
2a13a9b    host            host    local
gv39nyf    ingress         overlay swarm
ut1oc58    ty              overlay swarm
57a9ce7    none            null    local

Step 2: Deploy Initial GitLab

For LDAP Integration, obtain a read-only Service User (with the name: GitLab Integration) from LDAP Administrator.
For Slack Integration, obtain Slack Hook URL and Channel from Slack Administrator.
For JIRA Integration, obtain JIRA Integration user & pass from JIRA Administrator.
Create a gitlab directory and put a docker-compose.yml file with content, as seen below 👇🏻.
Create /data/gitlab to store data files and /var/log/gitlab to store logs.
We selected to install GitLab EE, because we may buy EE in the future. For details check the comparison table and EE explanation.

GitLab docker-compose.yml

#Deploy GitLab
docker stack deploy -c docker-compose.yml gitlab

Step 3: Important lines of the docker-compose.yml file.

Line 9: GitLab URL, that you login with LDAP credential.
Line 11: GitLab Container Registry URL, that you login with LDAP.
Line 14: Enabling LFS, to store large files in the repository.
Line 16: Set, Checking Scheduled CI/CD Jobs every 10 minutes.

After ~2 minutes, GitLab and Container Registry are ready. Please, check the URL and Sign-in with root user credentials. 👊🏻

Step 4: Prepare repository & API access credential.

Create a project with the name gitlab.

Add all files from https://gitlab.com/olgac/gitlab repository to your gitlab repository.
Create a full-powered GitLab Access Token from this URL for accessing API to manage GitLab by GitOps methodology.

Step 5: Install Runner

Run gitlab-runner.sh script to create a runner with tag “gitlab”. The script installs required executables and registers a runner to runner node.
There are some Types of GitLab Runners, for different purposes. We are using 🐳 Docker Executor Runners. Because it runs the job in a docker container and dies. This feature gave us the power to execute any type of executables and commands with the runner, by setting image property. 💪🏻 (.gitlab-ci.yml Line 7 & Line 16 ).

Each Runner can be in one of the following states and/or belong to one of the following types:

Disable Default to Auto DevOps pipeline for all projects by unchecking on URL http://gitlab.example.com/admin/application_settings/ci_cd. Because we will manage our pipelines with .gitlab-ci.yml

Step 6: Prepare .gitlab-ci.yml

With given .gitlab-ci.yml, developers can easily manage stages and jobs as-seen below.

.gitlab-ci.yml

After, registering Docker Executor Runner, we can run the first job of the pipeline to set-up permissions regarding permissions.csv.
The second (gitlab) job is waiting for a manual trigger to re-deploy Gitlab Server with given parameters and version.
The tags property is very important to select which job will run on which runner. If runner and job tag matches, the job executes on this runner. We are setting tag value while registering the runner in gitlab-runner.sh script via --tag-list property.

Step 7: Run Pipelines

After, Running (👆🏻above ) Setup Permissions Job; Users, Groups, Slack and JIRA Integrations should be created.
Our Best Practice for arranging groups, per sub-team, will have a Slack Channel and a JIRA Project.
As seen in permissions.csv file, each user has access levels to groups.

Access Levels `10=Guest 20=Reporter 30=Developer 40=Maintainer 50=Owner`

Step 8: Login, Build & Push 🐳

Also, GitLab Docker Registry is available to push images with given commands.

#Login Docker Registry via GitLab Internal Registry User
~$ docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY#OR, Login Docker Registry via Your LDAP User
~$ docker login registry.example.com:5005
Username: ahmet.olgac
Password: *******Login Succeeded#Push Builded Docker Image
~$ docker push $DOCKER_IMAGE

First, you should add registry.example.com:5005 as an insecure registry record for Docker Daemon for clients. You can add an SSL Termination Layer in-front-of the registry (for example: NGINX 443 to 5005), to work with the SSL secure registry.
For Mac OS, as seen screenshot below.👇🏻 For Ubuntu as seen script below.

#For Ubuntu
~$ echo "{\"insecure-registries\":[ \"registry.example.com:5005\"]}" > /etc/docker/daemon.json~$ service docker start

Then, login with LDAP user & pass to GitLab Docker Registry with the given command below👇🏻 for push images to here.

~$ docker login registry.example.com:5005
Username: ahmet.olgac
Password:Login Succeeded

Then, you can build and push 🐳 Docker images to the registry.
In addition, GitLab provides very useful predefined environment variables that accessible at job runtime. For example, check out the below sample.

Conclusion: Benefits with GitLab by GitOps Methodologies

Our cycle time is reduced, everything in one place (Cycle Analytics)
Simplicity & Focus on the core domain.
CI/CD pipelines simplified and easily maintained.
Saving team from the complex toolchains & hard integration problems.
Gained maintainable & manageable infrastructure.

PS: This article is the 1st part of the (5 articles) series, which digs Centralized Logging for K8S, Observability, APM for Cloud-Native Apps and Container Registry Topics with GitOps methodologies.