Continuous delivery and deployment with GitLab CI
This is a topic close to our hearts here at Kontena, Inc. The reason is that we’ve been practicing and doing this for years, even before Kontena, and everything we do now gets automatically shipped. I’ve also written before on some of the more conceptual considerations that should be taken into account when implementing continuous delivery and deployment capabilities.
In this article, I’ll take things to a more concrete level and walk you through how to setup a successful deployment pipeline using containers, GitLab, GitLab CI and Kontena.
Note that all the concepts and steps apply equally regardless of the CI/CD platform you are using. Naturally you will need to adjust the configurations of each step, but for most of the CI/CD tools out there, the setup is pretty similar.
I’ve built a complete sample workflow with our beloved Todo sample application.
Why a pipeline?
To summarise the earlier post, having an automated pipeline for delivering software gives you the ability to push changes to production environments faster and safer than doing it manually. By being able to push changes out faster and safer, you also have the possibility to go faster than your competition.
A quick word about the tools I’ll be using throughout this article.
Containers
I’ll be using containers as the fundamental thing with which we ship software. Containers give you the ability to package and run your application in a fully portable manner. With containers as the build artefact within your pipeline, you can ensure that in every step of the pipeline there is zero variance of the app. And containers make the actual deployment process so much easier.
GitLab & GitLab CI
For running the actual pipeline I’ll be using GitLab CI. It offers an easy to use platform for defining a pipeline configuration in yaml. And, surprise surprise, I like that it can run containers as part of the pipeline too. Being able to use containers as part of the pipeline, makes it very easily configurable as you do not need to go and install some random plugins to be able to do something. For example, if I’d be compiling Java code, I do not need to go and setup JVM on my build machines as I can now execute all the steps in containers.
Kontena
Setting up and managing containers by themselves does not really make sense. Your life is so much easier when you have some container orchestration tool in use which manages the actual deployment process for you. It handles things like rolling deployment, application loadbalancing and container scheduling to name a few. Without one, you end up scripting many of these things yourself. And trust me, it’s a pile of scripts you really do not want to manage long term. In this article I’ll be using Kontena as the deployment target platform to run my app in containers.
We really have created the most easy-to-use and developer friendly platform to run containers on. And with Kontena Cloud, things get even easier to setup and manage as you get everything hosted by us. Oh, did I mention you get free credits when you sign-up, and you don’t need a credit card to start with? Enough to get you started and to even try out the examples shown in this post.
The pipeline
There are probably as many variants of the pipeline steps as there are companies and projects implementing those. And of course, the runtime platform (language runtime, frameworks, etc.) poses some limitations and requirements for the pipeline. What also highly defines the pipeline implementation is the version control flow your project is using.
The example pipeline I’ll be building is based on the learnings and patterns we’ve been using to ship our own production services in the past few years. We’re using a slightly simplified version of the GitFlow workflow with many of our microservices, so I’ll base this sample project also on the same foundation.
Feature branches
All the development happens in feature branches. For each feature branch, each commit actually gets deployed to a testing environment. The deployment is done in such a way, that for each branch, there’s actually a different deployment happening. This pretty much means that for each branch, there’s constantly an application version running to be used by your QA folks for example. This also makes sure that everything is actually deployable since you’ll be able to fix deployment related issues in your feature branch already.
Master
In this case, I’ll use the master
branch as the constantly shippable product. Whenever we merge a feature, or any other branches into the master
we'll make an automated deployment to the staging environment. It ensures that your master
branch is always deployable and gives you a stage to test things out before shipping them into production. I'd strongly suggest using master
, or the equivalent of it, to be a protected branch into which you cannot make direct commits. So your commits are basically PR merges with appropriate automated testing, code review, etc. done.
Tags
We’ve been happy users of tags, a.k.a. releases in both GitHub and GitLab, to make production deployments. Of course the same things could be done using a pre-defined branch also. We’ve been using tags since it gives you also a pointer in time — the tag itself — to which you can easily refer to.
Environments
GitLab CI supports a concept called the environment. With this you can “bind” different environments into the stages and steps in your pipeline. With these you can for example, scope the secrets to be injected into the pipeline steps, or you can automatically shutdown a specific deployment for a given feature branch. In the example, I’ll be using three different environments to ship my application to.
When using Kontena, you usually have separate platforms for each of the following environments. Of course test
and staging
are probably running with less capacity and availability that your production platform. In practice this means that you could run the test
and staging
as mini
platforms, with only a couple of worker nodes connected. For your production
platform you want to have more availability built in, so you should always use the standard
version. That gives you a highly available setup for the management nodes (a.k.a masters), with clustered databases and everything distributed across multiple availability zones.
Test
For each feature branch I want to have the developer automatically seeing the feature being actually deployed and running. This gives more confidence that the further deploys will also work and also gives you a constantly running app to test and play with. It’s then easy to make, for example, a UX change and actually show the change running.
The test environment actually consists of multiple deployments as each feature branch gets deployed into the same platform. To make this really easy, I’ll use some advanced Kontena Stack techniques to easily customize each deployment a bit.
Staging
The staging platform always runs the application version on the master
branch. A new deployment is triggered for each commit into the master
. If the master
is a protected branch, each commit usually means a merge of some PR. Each deployment updates the existing deployment on Kontena and thus always makes the latest merged features available in staging.
Production
When ever you want to make a production deployment, you’ll do it by creating a tag. Creating a new tag triggers the pipeline and selects a set of jobs that will go and build the needed container(s), push them to the registry and make a deployment to the production platform.
Integrating Kontena platform to GitLab CI
By now it’s pretty clear that we’ll want to integrate our GitLab CI pipeline to make deployments on our Kontena Platform. There’s pretty much two different ways to do it:
- Connect to the platform master using its REST API
- Use Kontena CLI “embedded” in the CI pipeline
We usually use the latter method as it’s much easier and manageable.
The CLI can be easily configured to connect to a platform using environment variables. In order to do that you need to figure out a couple of things.
Platform URL
In order for the CLI to connect to the correct platform master we need to give it a URL to the master’s REST API. One of the easiest ways to figure that out is to check it from you local terminal when your own CLI is connected to it:
$ kontena master current jussi/demo https://bold-grass-3288.platforms.us-east-1.kontena.cloud
So for CLI to connect to this specific platform master we need to populate KONTENA_URL
environment variable with a value of https://bold-grass-3288.platforms.us-east-1.kontena.cloud
.
Platform Grid
We need to tell the CLI also which platform grid we want to operate. Again, it’s easy to check with your local CLI tool:
$ kontena grid ls
NAME NODES SERVICES USERS
test * 3 0 2
The one with the *
next to it is the one your local CLI is connected to. So we need to populate KONTENA_GRID
environment variable with the value of test
in this case.
Token
All the operations at the platform master API are authenticated and authorized using Oauth2 Bearer
tokens. So for the pipeline using the CLI we need to have a token for it. Use your local CLI to generate a new, never expiring token for the pipeline to use:
$ kontena master token create -e 0
Grab the shown access_token
and place it in the KONTENA_TOKEN
env variable for the CLI to grab it within the pipeline.
The jobs
With GitLab CI, as with most of the CI/CD tooling, you define your build and deployment configuration using yaml syntax. In GitLab CI you define the configuration in a file called .gitlab-ci.yml
placed on the projects root.
All the steps in my pipeline are executed as Docker containers. GitLab runner is set up so that it executes each step in a separate container and thus you get nicely isolated builds.
The steps executed depend on the trigger of the pipeline and are grouped together in stages. All the jobs within a stage are executed in parallel and the pipeline moves from one stage to another only after each job in a stage has executed successfully.
In this case I definitely want to execute different jobs based on the branch the commit was made on or if it was a tag that got created.
In the example project I’ve defined the following stages:
- test
- build
- tag
- deploy
- smoke-test
The default stages, if you haven’t defined your own stages, are: build
, test
and deploy
. I've changed the order of build
and test
since I'm using a Ruby based app that doesn't need actual building/compilation.
I’ve used a common variable to make things more controllable and to avoid repetition:
variables:
GL_IMAGE: registry.gitlab.com/jnummelin/todo-example:$CI_COMMIT_SHA
SMOKE_IMAGE: images.kontena.io/jussi/todo-example:$CI_COMMIT_REF_SLUG
SMOKESTACK: tododev-$CI_COMMIT_REF_SLUG PROD_IMAGE: images.kontena.io/jussi/todo-example:$CI_COMMIT_TAG
Let’s take a look at the jobs that happen for each stage.
Test
In the test
stage we're mostly interested in executing some local tests to make sure the application is even remotely possible to push towards production. As mentioned, the example app is a Ruby app so I'll use standard a Ruby environment to execute the basic unit tests.
rspec:
stage: test
image: ruby:2.3
services:
- mongo:3.2
variables:
MONGODB_URI: mongodb://mongo:27017/todo_test
script:
- bundle install --path=cache/bundler
- rspec spec/
stage: This job is only executed in the test
stage.
image: Use the standard Ruby Docker image to execute the job
services: What other services needs to be up-and-running during this job. In this case I'll need MongoDB running as the local tests use it. These are automatically linked to the build container.
variables: What variables are injected into the build container. In this case I'll tell the app where it can find MongoDB.
script: What is actually done during the job. In this case I'll first need to install all the dependencies using bundler and after that execute the actual tests.
That’s it, really. I didn’t have to go and install random plugins to be able to run Ruby jobs or anything. Nice.
As this is the only job in the test
stage, we'll move to the build
stage once the rspecs pass.
Build
The build stage basically builds me the container that I can use in further stages.
build-image:
stage: build
image: docker:latest
services:
- docker:dind
script:
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.gitlab.com
- docker build -t $GL_IMAGE .
- docker push $GL_IMAGE
stage: This job is only executed in the build
stage.
image: Uses docker:latest
image that has all the needed docker cli tooling built-in.
services: As I'm building Docker images, I need to have Docker daemon available. dind
actually refers to Docker-in-Docker, so the Docker daemon used in this job is actually running in a container.
script: Login to GitLab registry, build the "local" image and push it.
Tag
I defined a special tag
stage as I want to be able to tag the image as a separate step from the image building. As I want/need to also tag the images differently based on the branch/tag the pipeline is executing on it is good to have it as a separate stage.
There’s actually two different jobs defined for the tag
stage.
tag-smoke:
stage: tag
image: docker:latest
services:
- docker:dind
script:
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.gitlab.com
- docker pull $GL_IMAGE
- docker login -u "$IMAGES_KONTENA_IO_USER" -p "$IMAGES_KONTENA_IO_PASSWORD" images.kontena.io
- docker tag $GL_IMAGE $SMOKE_IMAGE
- docker push $SMOKE_IMAGE
only:
- branches
except:
- master
tag-smoke
the job basically takes the existing image built in build-image
job and tags it with the branch name. So for example, if my branch name is feature/cool-feature
it would tag and push image images.kontena.io/jussi/todo-example:feature-cool-feature
. GitLab CI automatically "slugifies"* the branch names so that they can be easily used in many places. This job is only run for branches where the branch name != master
.
*) Slugification:
Commit reference lowercased, shortened to 63 bytes, and with everything except 0–9 and a-z replaced with -. No leading / trailing -. Use in URLs, host names and domain names.
Naturally I need to provide the Kontena Image Registry login details as secret variables for the GitLab CI. For creating the needed token, consult the image registry docs.
To tag the image for production, the job is pretty much the same, except for the actual tag:
tag-prod:
stage: tag
image: docker:latest
services:
- docker:dind
script:
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.gitlab.com
- docker pull $GL_IMAGE
- docker login -u "$IMAGES_KONTENA_IO_USER" -p "$IMAGES_KONTENA_IO_PASSWORD" images.kontena.io
- docker tag $GL_IMAGE $PROD_IMAGE
- docker push $PROD_IMAGE
only:
- tags
In my case, I want to tag the image with the git tag so that it’s obvious to everyone which version of the app is running at any given point. And as I’m doing production deployments from tags only, this job is restricted to only tags.
Deploy
Now we have the image built and pushed to a Kontena hosted image registry. That’s kinda the first step of getting the application up-and-running. The second step is to instruct Kontena to go and deploy the app with the new image.
As with tag
stage, the deployment is naturally split into different jobs based on the target environment.
deploy-smoke:
stage: deploy
image:
name: kontena/cli:latest
entrypoint: ["/bin/sh", "-c"]
environment:
name: review/$CI_COMMIT_REF_SLUG
url: http://$CI_COMMIT_REF_SLUG.todo-testing.kontena.works
on_stop: stop_smoke
variables:
KONTENA_URL: $KONTENA_SMOKE_URL
KONTENA_GRID: $KONTENA_SMOKE_GRID
KONTENA_TOKEN: $KONTENA_SMOKE_TOKEN
VHOSTS: $CI_COMMIT_REF_SLUG.todo-testing.kontena.works
CI_COMMIT_TAG: $CI_COMMIT_REF_SLUG
script:
- kontena stack install --name $SMOKESTACK || kontena stack upgrade $SMOKESTACK
only:
- branches
except:
- master
deploy-smoke
the job defines a dynamic environment based on the branch name it is triggered on. I'll use a shared test platform to deploy all branches into so that needs a couple of Kontena Stack variables in my application stack:
variables:
release:
type: string
from:
env: CI_COMMIT_TAG
vhosts:
type: string
from:
env: VHOSTS
With these I’m easily now able to deploy multiple different branches into the same platform, just using different names for the stack and different variable values.
Assuming I’m working on a branch named feature/cool-thing
, this job would go and deploy my application stack with the name tododev-feature-cool-thing
to my common testing platform. When making the deployment, we'll be using an image tagged with feature-cool-thing
to deploy the application and the platform loadbalancer is configured to forward all traffic with the domain feature-cool-thing.todo-testing.kontena.works
to this specific application version. So automatically we get a branch specific sample app up-and-running.
But how are these arbitrary and dynamic DNS names actually pointing to something that could be actually accessible? It is possible to do some kind of wildcard DNS entries using CNAME
aliasing. What I've done is the following:
- define proper DNS A records to map my test platform loadbalancer(s) public IP address(es) to a domain name. For example IPs
1.2.3.4
and2.3.4.5
mapped to nametest-platform.kontena.works
- create a wildcard CNAME
*.todo-testing.kontena.works
that points totest-platform.kontena.works
As a result now foo.todo-testing.kontena.works
and bar.todo-testing.kontena.works
, and so on, magically point to the same public IP addresses and the Kontena Loadbalancer running there can go and proxy the traffic to the correct stacks based on the incoming hostname. Pretty neat.
The dynamic deployment from any branch gives a nice and super easy way to see the branch live. When we use these dynamic environments in GitLab CI we can even see these in the merge requests:
As you probably guessed, the staging and prod deployment is pretty much the same configuration.
deploy-stack:
stage: deploy
image:
name: kontena/cli:latest
entrypoint: ["/bin/sh", "-c"]
environment:
name: production
url: https://todo-demo.kontena.works
variables:
KONTENA_URL: $KONTENA_PROD_URL
KONTENA_GRID: $KONTENA_PROD_GRID
KONTENA_TOKEN: $KONTENA_PROD_TOKEN
VHOSTS: todo-demo.kontena.works
script:
- kontena stack install || kontena stack upgrade todo
only:
- tags
We just change the target environment, and the needed secrets for that, and limit the triggering only to tags for the production deployment and to the master branch for the staging deployment.
There’s one special type of job in the deploy
stage called stop_smoke
.
stop_smoke:
stage: deploy
image:
name: kontena/cli:latest
entrypoint: ["/bin/sh", "-c"]
variables:
KONTENA_URL: $KONTENA_SMOKE_URL
KONTENA_GRID: $KONTENA_SMOKE_GRID
KONTENA_TOKEN: $KONTENA_SMOKE_TOKEN
GIT_STRATEGY: none
when: manual
environment:
name: review/$CI_COMMIT_REF_SLUG
action: stop
script:
- kontena stack rm --force $SMOKESTACK
only:
- branches
except:
- master
It’s basically a “hook” which can automatically go and stop the branch specific dynamic environment. As we’re now running these dynamic envs in the same testing platform as the separate stack, stopping the env is pretty straight-forward. We just go and completely remove the branch specific stack installation. This again ties into the environments on the GitLab side nicely:
Smoke test
The last stage and job is to execute automatic UI smoke testing which is executed only for feature branches.
smoke-test:
stage: smoke-test
image: ruby:2.3
environment:
name: review/$CI_COMMIT_REF_SLUG
url: http://$CI_COMMIT_REF_SLUG.todo-testing.kontena.works
script:
- bundle install --path=cache/bundler
- curl -L https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-2.1.1-linux-x86_64.tar.bz2 | tar -xj -C /usr/bin --transform='s,.*/,,' phantomjs-2.1.1-linux-x86_64/bin/phantomjs
- rspec integration-spec/
only:
- branches
except:
- master
The smoke tests use capybara, poltergeist and phantomjs to execute testing through the application web UI, in headless mode. The tests are written in Ruby so I’ll execute them using the standard ruby:2.3
image. Phantomjs needs binaries in place so I'll go and grab them dynamically before running the actual tests.
How the smoke tests work and actually test the UI is worth its own post that I’ll publish in the near future.
Using containers in your build and deployment pipeline makes so much sense. Not only to actually deploy your own software but also as the basic execution environment for the actual pipeline. When you combine that with a super easy-to-use container platform like Kontena, you can have your pipeline running within hours. When using these kinds of automated deployment pipelines you’ll notice also one thing pretty soon: You don’t feel comfortable pushing manually, your pipeline is the only thing you now trust to do deployments.
If you want to try how easy it can be to ship your applications using automated pipelines, go and sign-up to Kontena Cloud, spin up your test platform and try the sample project for example by forking it.
Image Credit: Architecture Industry by Michael Gaida.
Originally published at blog.kontena.io on January 16, 2018.