CI/CD with Gitlab & Kubernetes

Today, after few experimentations, I’m able to show you how to deploy a complete CI/CD Pipeline based on Gitlab and Kubernetes with multi-environments and auto-deployment in the cluster.

To make this configuration at home, you’ll need a working Kubernetes cluster with two namespaces (default / dev) and a working Gitlab server with a runner dedicated to the proper git repo. You can use the DockerHub to push your Docker images but I prefer to use Gitlab’s internal registry.

The example website is called “Tropical Hosting” (a hosting company example website). To make it work we need this structure in our git repo :

• .gitlab-ci.yml (used by gitlab for manage our CI)
• Dockerfile (used by Docker to build our application
• deploy.sh (used by CI to deploy our application in Kubernetes)
• template/*.yml (our kubernetes resources like deployment, service, ingresses…)

In my case, I use two environments (dev|prod) in two different virtual-hosts handled by Nginx Ingress Controller (a reverse-proxy for Kubernetes) with TLS termination.

We will start with a Dockerfile :

FROM debian:jessie
MAINTAINER NGINX Docker Maintainers “docker-maint@nginx.com”
ENV NGINX_VERSION 1.11.9–1~jessie
RUN apt-key adv — keyserver hkp://pgp.mit.edu:80 — recv-keys 573BFD6B3D8FBC641079A6ABABF5BD827BD9BF62 \
 && echo “deb http://nginx.org/packages/mainline/debian/ jessie nginx” >> /etc/apt/sources.list \
 && apt-get update \
 && apt-get install — no-install-recommends — no-install-suggests -y \
 ca-certificates \
 nginx=${NGINX_VERSION} \
 nginx-module-xslt \
 nginx-module-geoip \
 nginx-module-image-filter \
 nginx-module-perl \
 nginx-module-njs \
 gettext-base \
 && rm -rf /var/lib/apt/lists/*
# forward request and error logs to docker log collector
RUN ln -sf /dev/stdout /var/log/nginx/access.log \
 && ln -sf /dev/stderr /var/log/nginx/error.log
EXPOSE 80 443
COPY website /usr/share/nginx/html
CMD [“nginx”, “-g”, “daemon off;”]

As you’ll see, I copy my static website from the website/ path and add it in my Nginx Docker container. So every modification in the code-base create a new dedicated Docker image.

Here is my .gitlab-ci.yml file with two major steps :

- build (build and push latest image with <branch>:<commit> tag)
- deploy (in dev|prod using kubectl and my deploy.sh script)

variables:
 DOCKER_DRIVER: overlay
 IMAGE_NAME: “gilabregistry/user/tropicalhosting”
build:
 image: docker:latest
 services:
 — docker:dind
 stage: build
 cache:
 key: “builder”
 paths:
 — ./.build
 script:
 — docker version
 — docker build — pull -t “$IMAGE_NAME:${CI_BUILD_REF_NAME}_${CI_BUILD_REF}” .
 — docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN gitlabregistry
 — docker push “$IMAGE_NAME:${CI_BUILD_REF_NAME}_${CI_BUILD_REF}”
k8s-deploy-Dev:
 image: lwolf/kubectl_deployer:latest
 services:
 — docker:dind
 stage: deploy
 script:
 — kubectl config set-cluster my-cluster — server=”$K8S_URL” — insecure-skip-tls-verify=true
 — kubectl config set-credentials admin — token=”$K8S_TOKEN”
 — kubectl config set-context default-context — cluster=my-cluster — user=admin
 — kubectl config use-context default-context
 — kubectl get cs
 — /bin/sh deploy.sh ${CI_BUILD_REF_NAME}_${CI_BUILD_REF} dev dev.tropicalhosting.com
 environment:
 name: dev
 url: https://dev.tropicalhosting.com
 only:
 — dev
k8s-deploy-Prod:
 image: lwolf/kubectl_deployer:latest
 services:
 — docker:dind
 stage: deploy
 script:
 — kubectl config set-cluster my-cluster — server=”$K8S_URL” — insecure-skip-tls-verify=true
 — kubectl config set-credentials admin — token=”$K8S_TOKEN”
 — kubectl config set-context default-context — cluster=my-cluster — user=admin
 — kubectl config use-context default-context
 — kubectl get cs
 — /bin/sh deploy.sh ${CI_BUILD_REF_NAME}_${CI_BUILD_REF} default tropicalhosting.com
 environment:
 name: production
 url: https://tropicalhosting.com
 only:
 — master

There is 3 code blocks, the build one, and two for the deployment, one for dev and one for prod. So you can easily add environment (pre-prod, testing…) by adding a git branch and a deploy block.

I’m using two variables in this file from Gitlab repo’s variables :
- $K8S_URL for access to the k8s API
- $K8S_TOKEN the token for admin user

After this, I have my k8s files in the templates/ directory (for example, deployment.yml) :

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
 name: tropicalhosting
 namespace: ${NAMESPACE}
spec:
 strategy:
 type: RollingUpdate
 rollingUpdate:
 maxSurge: 1
 maxUnavailable: 0
 template:
 metadata:
 labels:
 app: tropicalhosting
 spec:
 terminationGracePeriodSeconds: 60
 containers:
 — image: gitlabregistry/user/tropicalhosting:${BUILD_NUMBER}
 name: tropicalhosting
 imagePullPolicy: Always
 ports:
 — containerPort: 80
 livenessProbe:
 httpGet:
 path: /
 port: 80
 initialDelaySeconds: 30
 timeoutSeconds: 1
 imagePullSecrets:
 — name: registrypullsecret

You can add some ressources depending on how your application work. In my case I have one deployment, one service and one ingress (TLS).

Don’t forget to create your Pull Secret for your private registry if you don’t use the DockerHub public registry.

As you’ll see, my k8s *.yml files contains some variables. For parse them and replace with the correct value (image tag, vhost for my ingress…) I use a deploy.sh script (with the great envsubst tool) :

#!/usr/bin/env bash
TAG=${1}
NS=${2}
URL=${3}
export BUILD_NUMBER=${TAG}
export NAMESPACE=${NS}
export ENV_URL=${URL}
for f in templates/*.yml
do
 envsubst < $f > “.generated/$(basename $f)”
done
kubectl apply -f .generated/

It is quite simple, it takes 3 arguments, the image tag (<branch>:<commit>), the k8s namespace and the vhost URL.

All the k8s resources applied to the cluster are created in the .generated/ folder. I included it emply in the git repo. You can put all the resources you application needs in the templates/ folder and they will be parsed via envsubst.

With this setup, you’re able to commit in your dev branch, wait >1mn to see the result of your dev environment. If all your changes are correct, you can easily merge your changes to master branch to deploy them in production.

Pipeline’s builds

Pipeline’s environments

As you can see, I use one pod for myDev environment, and my production environment is scaled at 5 pods :