Creating a DevOps toolbox for Gitlab-CI

5 min readJan 20, 2020

I love GitLab, really do, and when I started to work with Gitlab and GitlabCI in 2016 every project that we have to use it I struggle with some basic things and these are the things that we are going to cover, things like lack of plugins and re-usability of your CI Yaml.

Gitlab DevOps Booth @ AWS Re:Invent 2019

So let’s talk about what we are going to do because GitlabCI doesn’t have plugins even a curl inside the Yaml can be a pain to write for several reasons, escape of special characters, double and single quotes, concatenation, etc.

So in this article, we are going to present a simple toolbox idea(I was inspired by DPL which I also encourage you to do the same process that we are going to talk here), it’s a pretty simple idea that can help you a lot with complex pipelines and re-use of code inside the pipelines, instead of writing everything inside your .gitlab-ci.yml we are going to create a toolbox to perform some sort of actions.

The problem

If we want to create a release inside of our Gitlab and we are using Community edition we would have to do something like this:

gitlab-ci.yaml

Yeah, that’s not pretty right? And guess what? It won’t be going to work without some effort to escape and make the quotes right, this is going to be a pain to fix it and another huge problem that, all of your pipelines would need to copy and paste this horrible Yaml.

Creating the solution

So how are going to make this more readable and easier for us? Let’s see, you are going to need:

Python 3.8.1
Docker
Appropriate libraries

Let’s start with basics, how do we create a new release with Python? Yep using the Python-Gitlab wrapper, so let’s start!

Before you start to code, you must set up your environment and do a:

pip install python

Then creates a new python script called create_release.py, this first snippet will be creating the Gitlab object and then support function to get the project ID, this will be useful for our release creation.

Now that we have our support function, let’s create the create release function.

Ok, this is not perfect must work but this code has a lot of problems, such as hardcoded information, re-usability, etc. But let’s see how this is going to look if we use it on our gitlab-ci-yml:

Ok, that’s so much better, right? but let’s make it even better on the next step.

Click!

Click! Click is a package that allows you to create CLI from python scripts.

Click implementation will allow us to use parameters/options on our script in a very easy way, so for this function, we will be receiving the Project ID directly from a pre-defined Gitlab-CI variable and the tag from git, for this article I won’t be covering the CI flow or how to generate the tag so I will just assume that you have your ways to creating and defining tags.

So let’s get started:

This is a sample of Click implementation:

Alright so let’s use this example to make our create_release function a CLI command:

This should do the trick, the only difference from the example is that we are using arguments instead of options, arguments are required.

Ok now our CLI is implemented, but how it looks on our gitlab-ci.yml?

Alright, that’s so much better, now we have a command-line python script and we are using predefined variables to create our release!

But wait, you said that we’re going to re-use this and right now this python script needs to be inside all of our repositories, you have lied to me!

Calm down, there’s a final catch!

Toolbox creation!

Remember that one of the requisites for this was Docker? Yeah, that’s right we are going to create our runner that will serve as a DevOps toolbox, it will have all of our scripts at /usr/local/bin/ so we can just invoke them!

So you will need a brand new docker file to serve as a runner, I’m using here the python official Image.

I would advise you to create a new repository with your runner and the python scripts, the structure will be something like this:

Yes, you should have CI for your runner Image as well :)

Now for the Docker file keep it simple:

Build the image, push it to your preferred registry, now you can simply do this in your initial .gitlab-ci.yml:

So, what changed?

We have moved the create_release python script to our DevOps tools repository and removed it from the project that we were working on.
Now we assign this specific stage to be run inside our devops_toolbox image, which contains our create_release python script, when you do this we can invoke it directly because it’s on the path.
Script tag with our create_release python without the ‘extension’ (Remember always set the shebang!!!)

Now that’s a lot better and the best thing that will be re-usable from your container, so every stage that you need to execute it using the tag from your custom runner.

This is just an idea, you can create multiple binaries to do multiple actions, slack hooks, upload artifacts to Nexus/Artifactory, deploy binaries to your services, DPL, etc, then you will have a fully DevOps toolbox docker image to support your Continuous Integration and Delivery.

Summary

Instead of having duplicate scripts inside your .gitlab-ci.yml we created a python script inside our repo.
To make it more readable we modified the python script with Click turning it into a CLI.
To make it even better and re-usable we embedded it into a custom Docker image that will serve us as a runner inside the Gitlab-CI.

Things that I didn’t cover that you should look into:

Python tests
Error treatment, remember if your API returns 404 the script may return exit code 0, you must treat that.
Entire CI/CD flow, we just did a simple stage.

That’s it for now, hope it helps anyone using Gitlab-CI :)