Buildbot 1.8.0

I am pleased to announce the release of Buildbot 1.8.0.

The biggest news is that this is the last one to support python 2.7. We are not planning to make it an LTS release unless users with good reasons to stay on python 2.7 step up to help and maintain a stable branch. Next release will be 2.0 and we will drop support for python 2.7 and 3.4 in that release as other big python projects do. This major release will also remove some deprecated API (e.g. slave vs worker)

As the end of year came, we got sad news about Hyper.sh shutting down. This is sad as we think hyper.sh is good tech and we hope the folks behind it will find way to continue to grow their ideas. This is also sad for us because hyper.sh was hosting and sponsoring the CPU resources which supported our metabuildbot CI. We thank them a lot for that as this really eased the task of maintaining buildbot. We implemented pre-commit CI with this with a lot of coverage and a large configuration matrix. This allowed us to release once a month with lots of confidence on the limited human resources we have. Before that, we used free hosting services like travis, circle-ci, and appveyor (we still use them by the way), but those are limiting the number of parallel build you can make, and as our pipeline grew, the feedback time grew. Maintaining our own latent workers on hyper did help a lot to make all the tests in parallel to try and tend to achieve the 10–15min feedback time.

So we had to rebuild our latent worker environment. In 2019 the obvious choice is Kubernetes. Kubernetes provides a container based orchestration framework that is able to autoscale at various level. Kubernetes is open-source and all the industry is moving toward using or providing this API. Not all the hosting actors have the same level of cost efficiency in term of autoscaling, but I am sure this will progressively smooth out, and choosing Kubernetes is a guarantee of cloud agnosticity.

So we built for 1.8.0 a KubeLatentWorker, and we now run our Buildbot infra on GKE. This support is still experimental and we have some known bugs with fixes that were not able to make it in the release. This kubernetes support was initially contributed by Julien Girardin. Thanks a lot for working on it, this was very useful to have a working starting point.

Our strategy for this deployment is to use node autoscaling with big nodes (6 vCPU). This allows to limit the time downloading build container image. It is only downloaded once and then reused for other builds. The GKE autoscaling starts quite fast (3min cold start including image download), and then takes 10min to scale down (the timer is unfortunately not configurable). Compared to hyper, we have seen our build times decrease a little bit, but as we are still working with a trial account we can only scale to one node, and this is impacting our queue time a lot (the build time is increased because the latent worker’s pods are waiting for available CPU and memory)

You can find the complete release notes as usual in our documentation

Thanks a lot to our contributors for this release!

git shortlog --no-merges -ns v1.8.0...v1.7.0
    20  Craig Rodrigues
    18  Pierre Tardy
    17  Povilas Kanapickas
     1  13costermann
     1  Robin Jarry
     1  ciarancourtney
     1  Pavol Misik
     1  Chih-Hsuan Yen
     1  Curt J. Sampson
     1  Julien Girardin