Wait pipelines in GitLab

7 min readJun 26, 2022

For a few last years I work a lot with GitLab and can say with certainty that it becomes better and better every release, not like many other tools.

One of the best new features

For example triggering pipelines: it was only possible via API that is not a rocket science. This approach has a huge disadvantage: you trigger an external pipeline asynchronously and can’t even see if something goes wrong there.

What if you don’t need just to trigger another pipeline from the current one, but also wait for it to finish and check the status? Oh, it becomes like a monster task to implement then! You need to write a script that will check whether this external pipeline is finished every N seconds and return the final result via intermidiate file (the easiest way coming to my mind). Fortunately, somebody already shared a docker image called pipeline-trigger that can help you with achieving this quite easily! But it was still quite dangerous to use open source image for such approach.

In the beginning of 2020 GitLab moved Multi-project pipelines feature to all tiers and it became so easy just out of the box! This is a great example how other CI/CD platforms should implement it: super straightforward! There are still some little disadvantages comparing to Trigger API, but nice think to always try once needed.

trigger external project pipeline synchronously

A feature that’s still missing

Despite the above, there are still some missing features which (in my opinion) should be implemented long time ago. I want to tell you about one of such features that I needed to implement myself.

Sometimes we use dedicated runner for particular kind of jobs, e.g. for test automation pipelines like in my case. I need to create some docker containers in the runner from a pipeline with specific ports and I cleanup it in the end. So, I cannot run parallel pipelines as I cannot allocate the same port for different docker containers or my cleanup from one pipeline can kill some containers that are in use in a different pipeline. I think, it’s very easy example and quite a lot of people experience the same issues.

Does GitLab provide us with some workarounds? Of course, one of the first features you might google is called resource groups. It allows you to mark jobs with some resource that you want to block not put all the others jobs in queue until this temp resource is available. But it can only be used in stage level, not in pipeline level, so it doesn’t help in my case. You can see that such issue created years ago and still open.

In this simple example you can see that jobs marked with the same resource group will be running one by one, but it doesn’t guarantee that first you’ll have all jobs from the 1st pipeline finished before the next one is started.

Resource group cannot be used in pipeline level

The next solution that you can find is to go to GitLab Runner’s settings and set “concurrent” option in global level to “1”. Firstly, not everybody has this kind of access and even more so will want to do this kind of admin job. Secondly, we’ll change this settings for all the runners configured in this config.toml file. Thirdly, it only solve the same problem as resource group, so doesn’t help us at all. You can also check this opened issue for some more details.

The same goes for another popular solution in web to set “limit” option to “1” in a runner section. But it’s also only for concurrent jobs, not for pipelines.

But we can find quite a lot of tricky custom solutions like this one, but the idea is the same: use GitLab API to check unfinished pipelines and wait until we can proceed. None of them worked for me, so it’s time to…

Create my own bicycle

Let’s think what do we want to achieve.

For every new pipeline check that there are no other active pipelines and we’re safe to continue.
Identify what is active pipeline for us: in my case I should take into account all running pipelines and pending pipelines with unfinished jobs. The last one is not obvious: we can run this call when another active pipeline has just finished one job and created another one, but not finished - then its status will be pending for a few seconds/milliseconds.
Wait until there are no active pipelines other than current one and we can proceed with further jobs.
Block this stage with waiting mechanism with resource_group! Otherwise two pipelines can start at the same time and it will be too difficult to identify which one should start first.

For all concurrent pipelines I want to see the next picture:

Allow only 1 running pipeline with business jobs and 1 with wait job, the rest should wait for available resource

As for me, it looks like a really obvious feature that should be implemented in such a cool tool like GitLab CI! But okay, we have a plan, so let’s proceed.

Let’s script!

I decided to do it all in a raw shell script to make it easy to reuse for any language and to allow every engineer who is not a professional programmer to be able to understand what’s going on inside. Let’s try to use curl and jq only not to overcomplicate it with too many tools.

I’ll use a few environment variables here:

PROJ_ACCESS_TOKEN - required to make API calls. I’ll describe below how to get one in “Project configuration” section
CI_API_V4_URL - predefined variable in GitLab CI that is equal to https://gitlab.com/api/v4 in our case
CI_PROJECT_ID - another predefine variable that equals to current GitLab project id.

It’s very easy to get a count of currently active pipelines:

active_pipelines=$(curl --silent --noproxy '*' --header "PRIVATE-TOKEN:$PROJ_ACCESS_TOKEN" "$CI_API_V4_URL/projects/$CI_PROJECT_ID/pipelines?status=running")
active_pipelines_count=$(echo $active_pipelines | jq '. | length')

A bit more tricky to get all pending pipelines, make some for loop on them and check whether each has either failed or succeded jobs (that means pipeline is in progress, just preparing a new job while we’re sending curl request). There is a little problem as well: for shell loops we cannot use an array that jq returns, so we can use while and read through all the parsed jq pipelines:

pending_pipelines=$(curl --silent --noproxy '*' --header "PRIVATE-TOKEN:$PROJ_ACCESS_TOKEN" "$CI_API_V4_URL/projects/$CI_PROJECT_ID/pipelines?status=pending")
pending_jobs_count=0
echo "$pending_pipelines" | jq '.[].id' | while read -r pipeline_id; do
  finished_jobs_len=$(curl --silent --noproxy '*' --header "PRIVATE-TOKEN:$OMNIBUS_UI_TESTS_ACCESS_TOKEN" "$CI_API_V4_URL/projects/$CI_PROJECT_ID/pipelines/$pipeline_id/jobs?scope[]=failed&scope[]=success" | jq '. | length')
  if [ $finished_jobs_len -gt 0 ]; then ((pending_jobs_count=pending_jobs_count+1)); fi
done

Okay, we can wrap it into functions and check the total active pipelines count. Now we need to wait if we have more than 1 active pipeline (that is running our “wait” job) and check every N seconds (5 in my example) whether other active pipelines are finished

pipelines_count=$(get_pipelines_count)
printf "Currently $pipelines_count active pipeline$([ $pipelines_count -gt 1 ] && echo "s were" || echo " was") found\n"
until [ $pipelines_count -eq 1 ]
do
  printf '.'
  pipelines_count=$(get_pipelines_count)
  sleep 5
done

All I want to do in the end is to check total time execution and print it nicely with a format “…% minutes, % seconds”

Let’s store initial time before our script beging with “start_time=$(date +%s)” and finish it with:

end_time=$(date +%s)
total_time=$(( end_time - start_time ))
minutes=$((total_time / 60))
seconds=$((total_time - 60*minutes))
final_time=$(echo "$([ $minutes -ne 0 ] && echo "$minutes minutes, " || echo "")$seconds seconds")
printf "\nLooks like it's only 1 active pipeline at this moment. Proceeding after $final_time\n"

The full script you can find here.

Project configuration

You need to create access token in order to call GitLab API endpoints for your project. If you have ultimate license, it’s better to create project access token. Otherwise, you need to create personal access token. Only “api” scope should be enough for our case, so create token, copy it and put under your project > Settings > CI/CD > Variables

GitLab API access token (either personal or project one)

To initiate example GitLab pipeline that will use free docker runners, we should find a correct image. All we need is two tools: jq and curl. There is a nice little image existing in GitLab Registry that has both of them, let’s use it!

image: registry.gitlab.com/gitlab-ci-utils/curl-jq:latest

wait_queue:
  resource_group: wait
  script:
    - chmod +x wait-pipelines.sh && ./wait-pipelines.sh

So, we have a job execution our wait script and we need some other job that will be running for a while to check that it’s working. For full example reach out to my GitLab repo.