Introducing Gitlab Project Configurator

Align settings for hundreds of Gitlab Projects

TL;DR: This article presents Gitlab-Project-Configurator (GPC) proudly released under MIT License by Groupe Renault. GPC is a “configuration as code” tool used internally to align settings of hundreds of Gitlab projects and groups.

A git project that configures hundreds of other projects using GPC

Gitlab is a great tool to store your code and run Continuous Integration Pipelines on it. It works fine for personal projects and is also a great tool for corporations as their main repository for source code.

It is even greater at experimenting new ways of doing things, project creation is easy (and it should be easy).

But as project number grows, managing them becomes harder and harder. Doing everything in the UI is great, but things get a bit harder when you want to apply the same settings to every projects in a single group.

It would be much scalable to write down the wanted settings for projects (and groups) in files and apply them automatically, using the Infrastructure as Code principle (or more precisely, “Configuration as Code”).

Let’s start by an example

We want to apply this setting to a set of project in Gitlab.

Let’s say we would like for a specific group to set all their project with the setting “Fast-forward merge”. Not the whole Gitlab instance, not only a few projects. Maybe a few hundreds of them.

And with some exceptions, let’s say in the middle of all of our project we have one where we do not want this settings applied for some obscure reason.

We can manage to use the API ourself, thanks to the amazing Python-Gitlab module. But if we add another settings, a few other exceptions, and we will finally end up with a really hardcoded script, hardly maintainable, and that cannot be used on other situations.

Introducing Gitlab Project Configurator

When we started really scaling up our usage of Gitlab, we wanted to ensure that we had a way to align some project settings following the “as code” paradigm:

• configuration is described in a text file
• evolution of the configuration must be tracked like any other software change, ie, in Merge Requests
• tracking who did a change, when and why is extremely important.
  The git history provides it directly.

We tried to work with GitlabForm at the end 2018, but we were not able to do what we wanted with it:

• we want to define a set of logically-related settings in “rules” or “policies”
• several rules can be defined and inherit from each other
• rules are to be applied on some specific projects, or on all projects of a Gitlab group, or matching a regular expression.

Example of projects configuration and customization

Sometime a “generic” rule to be applied to a project needs some kind of customization.

Let’s say you want to configure 3 projects (P1, P2, P3) in a group with the following settings that all projects would follow:

• merge method set to fast-forward
• define a variable “MY_VARIABLE” with a specific value “MY_VAL”

But, one of the project (P3) is already using the variable “MY_VARIABLE” with another value “OTHER_VAL”, and changing may have side-effects.

So, we may want to apply these settings to P1 and P2, but on P3 to keep previous variable value for the moment. This becomes a “Technical Debt” which can be reimbursed later. This is an option often chosen when deadlines approaches.

That’s why we created Gitlab-Project-Configurator, allowing to apply precisely project and group settings, with lot of possible customization.

Rules are defined, and then applied to projects.

GPC is command line tool, written in Python, that will take a description of what you want to set on Gitlab’s projects and group, and apply as much as possible. Key features are:

• Security: all the configuration are centralized and it is easy to update credentials or password used in multiple projects.
• Traceability: If someone manually modify a setting manage by GPC, modified settings will appear in the GPC report.
• Reliability: GPC will apply settings defined in configuration files, and help to align multiple projects settings.
• Dry-Run: GPC can run in “dry-run” mode, for example in a merge request, and it will report what will be changed, but without applying the settings. This allow users to review and verify changes before applying the settings.
• Reports: GPC provides reports (HTML, email) of what has been done, and what failed, so post-mortem is easily done.

The main way to use GPC is by using the docker image provided by the project and run it directly on your project:

image: registry.gitlab.com/grouperenault/gitlab-project-configurator:latest

Note: This command will only work on gitlab.com server.

How it works?

Rules Definition

Rules are defined in one or several files.

Base rule

This is where you describe which settings your project should follow.

For example: all projects of my team should have a default branch named ‘master’, protected and only allowed to merge (no one can “push on it”), with all tags protected.

And so on.

Derivative rules

Then we can imagine a derivative of this base rule that adds a badge and define other rules.

Derivative rule

Apply rules on projects

Next step is to apply these rules on projects.

Rules applied on a project with customization

A special keyword named custom_rules allows to add exception to the rules you want to apply. It is probably a good idea to add comment on why this exception needs to be done in this case.

Our experience shows that when transitioning from a completely freely managed project to a project managed under GPC, we often faced situations where we wanted to do only parts of the migration at a given moment: either all variables cannot be migrated to GPC yet, or it raises some minor incompatibilities on some projects that would require manual changes.

But when migrating hundreds of projects at once, we were happy to use some carefully crafted custom_rules for a few days or weeks until this technical debt can be reimbursed while keeping or release schedules untouched.

Recommendations

We have a few recommendation when using GPC:

• you need to store your settings in a private project.
• Add a schedule to apply your change on a regular basis. We apply our change twice a day, so if someone alter some project settings under GPC, we know it will only last for a few hours before the settings are reset.

Project Information

Gitlab-Project-Configurator is released under MIT license starting by version number 3.0.0. We chose to maintain the main codebase in a private Renault repository, and perform an automated synchronization with the Opensource project on a regular basis. We do not have any Renault-specific features, only intensive integrations tests still heavily customized and some internal configuration files.

The open source version is automatically updated using a CopyBara script.

We are looking forward to contribution, and they will be merged internally first before being published back to open-source project, maintaining the original commit authorship.

You can have a look at a live demo project here.

The main documentation is accessible here.