Continuous delivery with GT Releaser
Releasing and deploying new versions on every commit
In this article we explore GT Releaser, the engine used by Glamorous Toolkit to build reproducible releases on every commit that keeps examples green.
GT Releaser is an engine for releasing deeply nested Pharo projects. It sounds like a trivial thing, but it’s an essential piece that allows you to develop trunk-based and release confidently on a push of a button, or two. Or a script for that matter.
The need for a release tool
Glamorous Toolkit is developed using trunk-based development. In short, the development branch is the master branch and it always contains the latest version of the projects. This helps to reduce the feedback loop between when a problem is introduced to when it is detected, as there are no long-lived feature branches, or long waiting times before integrating pull requests.
Nonetheless, this way of working raises challenges, specific to trunk-based development, the most important being how users find and load a stable version of the project, given that any new commit can potentially break the system. If the project were in a single repository with no external dependencies, this would have a simple solution: use the commit hash.
However, that is rarely the case. This is why we created GT Releaser.
Releaser is a solution to release versions for deeply nested Pharo projects spread across many git repositories with multiple loading configurations, that offers multiple strategies to create new versions and deal with external dependencies.
We use it for releasing Glamorous Toolkit itself: every commit where all examples pass results in a reproducible release that is also deployed.
In this article we explore the challenges that GT Releaser aims to solve, look into several usage scenarios, and go through the steps for building a release.
Challenges to overcome
Glamorous Toolkit does not reside in a single repository. Instead, as it consists of multiple individual projects, each project resides in its dedicated repository. This ensures that projects can be loaded individually if needed, and also evolve separately: a user can load the XDoc project for eXecutable Documents, or the gtoolkit-examples library for testing and documentation, without having to depend on the complete Glamorous Toolkit project.
Individual projects do not always have a single loading configuration. There can be multiple loading configurations for different parts of a project. In the case of gtoolkit-examples, the BaselineOfGToolkitExamplesEngine configuration only loads the core component for running examples as tests, while another one, BaselineOfGToolkitExamples loads the complete library.
The two configurations mentioned above are Metacello baselines. Metacello is a package management system used in Pharo. Metacello baselines contain the list of packages from a project, their loading order and their dependencies to external projects. GT Releaser works with Metacello baselines.
Glamorous Toolkit also has dependencies to external projects like SmaCC, PetitParser, or TaskIt. Not all of these dependencies are equal. Some, like SmaCC, are used deeply within the system and we always want to load the latest version. Others like TaskIt, have stable versions that change rarely.
The current version of Glamorous Toolkit loads code from 29 repositories using 43 loading configurations (Metacello baselines). Out of these, 16 repositories are internal to the project, 3 are for the graphical stack based on Bloc, and 10 represent external dependencies. The picture below show the dependency graph. Each rectangle is a repository containing baselines.
Usage scenarios
For highlighting usage scenarios let’s only focus on the Brick project and its dependencies. Brick is the widget library used in Glamorous Toolkit. It has dependencies to other projects from Glamorous Toolkit, like gtoolkit-example and gtoolkit-installer, as well as to external projects like Beacon and TaskIt.
Building a new release version for Brick requires creating new versions for all projects internal to Glamorous Toolkit and updating their baselines to load those versions, as well as loading fix versions for external dependencies.
The main strategy in GT Release is to perform a release by merging the development branch into a release branch, followed by creating a new tag on the release branch that uses a semantic version as the release number.
Currently Brick is at version v0.5.3. Creating a new release increments the patch number resulting in version v0.5.4. The version number is incremented only for repositories that have new changes. For example, the figure below shows the next release, v0.5.4. For repositories colored in grey no action is needed during the release. New versions will be created only in the Brick and Bloc repositories, as only they have new changes.
Loading the created release can then be done directly using Metacello.
Metacello new
baseline: 'Brick';
repository: 'github://feenkcom/Brick:v0.5.4/src';
loadIn case we want to create a new version in all repositories we can force one. This could be needed when we want to release a new minor or major version. For Brick, the release where we move to version v0.6.0 is shown below.
One observation about the previous release, is that even if we forced a new version, the repositories for TaskIt and Beacon are still grey. This is the case as they are external dependencies for which we do not want to create new versions. Instead in the release we need to point to an existing version.
When dealing with external dependencies, GT Releaser supports two strategies: (1) depend on a fix version available in the repository or (2) depend a commit from that repository. We can see this in the previous release. For TaskIt the release depends on version v1.0.1, already present in the repository. In the case of Beacon, the hash of the latest commit will be used.
While simple, with only these feature we can generate new releases for the entire Glamorous Toolkit on a daily basis.
Executing a release
Until now we just looked and the motivation behind GT Releaser and usage scenarios. Next let’s dive into how to create an actual release.
GT Releaser relies on an API for performing actions, and inspector views for the actual user interface.
There are four main steps for creating a new release:
- create a release configuration,
- create the exporter that can perform the release,
- execute the actions for creating the release, and
- execute the actions for pushing the new release.
First, we create a release configuration that indicates among other, the release branch, how we want the next version to be computed, and that describes how to handle external dependencies. The code snippet below specifies that:
- the release branch is named
release, - the next version is computed by incrementing the patch number,
- Beacon is an external dependency and we depend on the latest commit,
- TaskIt is an external dependency and we depend on the loaded tag, and
- by default no version is created if there are no new changes in a repository.
Second, we create the exporter that knows how to execute the release. For that we build first a model of the baseline, and then created the actual exporter based on our previous release configuration. The exporter object has a view that shows us the structure of the release.
Third, we perform the actual release locally. All actions executed when performing the release are modelled using objects. We can view the complete list of actions that create the release using another inspector view.
Last but not least, we push the local release to the remove repositories. This is also done through dedicated actions that we can view using the inspector.
This example only covers a single usage scenario, exemplifying the steps for creating a new release. More documentation about how to use GT Release is available directly in the image, as an executable document. The entry point is the class comment of BaselineOfGToolkitReleaser.
GT Releaser emerged from the need to enable continuous deployment and delivery for Glamorous Toolkit. Glamorous Toolkit offers an interesting case study both because it is made of deeply nested projects spread across multiple repositories, and because it has different kinds of external dependencies that need to be dealt with. At the same time, we rely on trunk-based development to reduce the need for long-lived branches and pull requests, and have a new version after every commit if all examples are green.
If it works for Glamorous Toolkit, it should work for your projects, too. Give it a try.
