Published in


Introducing glaball Open Source CLI tool to manage multiple GitLab instances

In the most common cases, having multiple GitLab installations is considered an anti-pattern. However, this is not always true: we manage dozens of our customers’ self-hosted GitLab instances. Periodically, the need arose to do something on all instances at once without wasting time repeating routine operations over and over. Part of that was monitoring critical GitLab updates. But there were other aspects as well including searching through the file contents in all installations or even creating users.

We came up with a solution to these problems called glaball. Originally created as part of our internal development effort, we are excited to make it publicly available now as an Open Source project.

Existing solutions

At the time, there had been several tools that had offered partially functionality of what we were looking for. Unfortunately, they could only operate within a single GitLab instance and did not support bulk requests to multiple instances.

We could use a script to automatically process all the instances in question, but what we wanted was an all-in-one, easy-to-use solution. To make it possible we considered the following options:

  • The excellent python-gitlab library with its own CLI utility;
  • The glab utility (GitLab CLI tool) — the equivalent of gh for GitHub. This one is written in Go;
  • The go-gitlab library — a client for the GitLab API used by the glab tool.

We decided on the second option because Go is faster and we are more used to it.


We outline the following general requirements for our tool:

  1. High performance regardless of the number of managed instances. That is, we had to make requests run in parallel and use the cache wherever possible;
  2. High speed for objects (users, projects, etc.) grouping and sorting; grouping and sorting by multiple fields;
  3. Minimal amount of code; active use of ready-made maintained libraries; the DRY principle.

As we mentioned above, our tool is based on the go-gitlab library. It is actively developed by a large community of nearly 400 contributors. We added wrappers for functions/methods running in parallel on all instances.

The following tools also helped us meet our requirements:

  • the peterbourgon/diskv file storage which is good for caching;
  • gregjones/httpcache — in our case, that is the easiest way to cache HTTP requests to the GitLab API. It is also the easiest to integrate;
  • ahmetb/go-linq — the .NET’s LINQ for grouping/sorting objects. We needed an SQL-like solution, but we didn’t want to use SQLite because of its extra dependencies, while another implementation had too many lines of code.

By putting all those building blocks together, we ended up with glaball. The CLI tool performs the necessary actions on the GitLab installations via the GitLab REST API.

glaball features

Initially, glaball was designed for checking if critical updates were available. The tool monitored the current version of GitLab and alerted us if a critical CVE was detected in the version of GitLab we were using. However, such monitoring was tied to our internal systems, so publishing glaball code “as is” would not be of much use to a wider audience.

Over time, glaball grew to include a number of other features that we often use in working with GitLab instances. Those reduced the time it took to perform certain operations and increased operation engineers’ efficiency. This updated tool became the basis of the sources we currently publish as an Open Source project.

1. Search by file content

We have added a function to search for specific content within files using regular expressions. This thing came in handy during the mass migration from werf v1.1 to v1.2. Here’s what it looks like:

glaball projects files search --filepath=".gitlab-ci.yml" --pattern="1.1"

2. User management

When a new engineer joins the team, you have to add their user to the respective GitLab installations. With glaball, it’s a breeze:

glaball users create --email=test@test.com --username=test-glaball --name="test glaball" --password=qwerty

In a similar fashion, you can use glaball to block, delete, modify, and search for users. Here is how to change a user’s password:

glaball users modify --by=username test-glaball --password=newpassw0rd

Note: There are more appropriate solutions like LDAP available for user management. However, they did not suit us for various reasons (we’ll leave those out of this article). On top of that, we wanted a universal tool…The command above prints a list of all projects that use werf v1.1. Consequently, we can now plan the appropriate task for the engineers and provide them with a list of projects to switch to v1.2.

3. Admin verification

glaball can also display lists of users grouped by various attributes. For example, the command below checks for redundant administrators in GitLab:

glaball users list --admins=true --group_by=username

… or administrators with two-factor authentication disabled:

You can automate these operations by implementing continuous monitoring with alerts being triggered when certain criteria are met.

4. Cleaning up the GitLab Registry and other schedules

We use the werf cleanup jobs as part of scheduled pipelines in the repositories in order to regularly clean up the GitLab Registry. However, sometimes one might just forget to. glaball is perfect for reviewing cleanup jobs:

glaball projects pipelines schedules --active=false

The command above ensures that the Registry does not eat up too much space. It is also a good candidate for automated monitoring of cleanups in all the repositories where werf is used.

Note: By the way, what does one do in the event errors occur, as shown in the screenshot above? First, trace them to their causes and eliminate them. Then, re-run the operation for a particular instance using the -f flag. If the error is due to exceeding GitLab API request rate limit (which is disabled by default), consider adding the rate_limiter.enabled parameter in the configuration of the instance in question.

glaball’s support for pipeline schedules including the feature of filtering by active/inactive status allows you to keep track of other scheduled repository stuff. For example, you can:

  • find out if deployment by schedule is used;
  • conduct various scheduled tests;
  • disable the test environments used for development at the end of the day (to save money).

5. Various operations

There command to display a list of repositories supports sorting, grouping, and filtering. For example, here is how you can print a list of archived projects:

glaball projects list --archived=true

… or a list of public repositories:

glaball projects list --visibility=public

The whoami command displays information about the current API user.

A few words about security

Note that all tokens in the glaball configuration file are stored in plaintext. That’s an acceptable compromise for our usage scenarios, but things may be different in your case.

Here are some related recommendations:

  1. To avoid token leakage, set the correct permissions on the configuration file and keep it in a safe place. By default, it is stored in the config.yaml file in the user’s home directory ($HOME/.config/glaball/config.yaml).
  2. If possible, use the read_api-scoped tokens and avoid tokens for GitLab administrative accounts (or store them in a safe place).
  3. Make sure that user tokens are time-limited to prevent attackers from using overlooked or lost tokens.


Throughout developing glaball, we gained valuable experience with Go parallelism as well as with GitLab and its API. We are going to carry on developing the project into the future.

Perhaps the biggest drawback is that there is no way to add new GitLab instances to the configuration through the command-line interface. (The glab tool is great at that by the way.) You have to manually add hosts and adjust their settings in the configuration file. glaball also lacks modular tests: none exist in any form at the moment.

If you have any ideas or suggestions for new features, feel free to submit them as issues (or even pull requests). We would also appreciate your stars on GitHub.

This article has been written by our software engineer Vasily Maryutenkov. Follow Twitter to get new excellent content from Flant!



Professional DevOps outsourcing services with a strong passion for Kubernetes.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store