Migrating to Bazel from Maven or Gradle? Part 2 - How to Decide on CI Server and remote execution / caching

There are two basic options when you look at the CI/remote execution landscape for bazel. On the one hand there is Google Cloud Build and on the other hand there are several OSS options.

Google Cloud Build (GCB) offering for bazel is actually comprised of 3 different products:

1. Container Builder — the CI server itself which runs each build step on its own docker container
2. Results Store (RS) — build results product that optimized for the bazel experience
3. Remote Build Execution (RBE) — a cloud worker farm that allows to run each build action on its worker

Build flow

The container builder has quite a simple build configuration design using yaml or json files to describe each build step. The basic premise is that each build step is run on its own docker container — which means there is much more control over the runtime environment and less possibility for dirty builds and tests.

For your builds steps, you can use pre-defined containers like git, or bazel and configure the arguments for them:

You can also create your own custom build steps that can run a script from source on top of a generic Linux container:

This is still a young CI solution which lacks a few important features such as builds status dashboard, the option to have pre-heated workers with custom containers, and auto-scaling of workers pool. But I know the good people at Google are working hard to deliver on these soon.

If you choose the OSS/DIY route then Jenkins is the consummate choice with a very mature product with much more flexibility and control over the build flow. The CI/CD pipeline is configured using a very powerful groovy DSL that allows you to define very complex build flows.

The biggest downside for using Jenkins, is that there is currently (Feb 2019) no ‘Bazel plugin’ for viewing/querying build results. More on this in the next section.

Build results

A very important requirement for Bazel’s ecosystem is an easy way to view and query the results of each build. Bazel is optimized to build the entire code repo with a single build command `bazel build //…`. And the output log can be quite massive and encompass the result of many many build and test targets. Which means it is very important to have a tool that dissects these results and allows to filter them easily.

Google has a solution for this issue in the form of Results Store (RS)

RS consumes the build events that Bazel outputs through the Build Event Protocol (BEP) and stores them for future querying.

Here is a screenshot of how a build result looks like in RS:

It allows to easily navigate through various build/test targets and view specific compilation errors and test logs. It allows to filter the results according to sub-areas of your repository.

Other services in your organization can consume build events and then query builds results through a dedicated API.

Results Store UI and API are also currently in Alpha.

There is currently no open source tool (e.g. Jenkins plugin) that has these abilities designed specifically for bazel build output.

Remote Execution

One of the most powerful features of bazel is the ability to run builds remotely on a worker farm, this allows for much faster, scalable builds and the ability to re-use the cached results by subsequent runs on CI and on developer machines which share the same platform (e.g. Linux).

How much faster? you can see below how remote execution and caching can dramatically decrease build times:

build times comparison, of a very large wix repository

There are several OSS projects that enable to run such a worker farm on the cloud or onPrem, among them are: Buildbarn, Buildfarm and BuildGrid. They are in various stages of development and maturity and will of course require dedicated management of the underlying machines.

Google is offering a hosted solution called Remote Build Execution (RBE) which is currently in Alpha.

Here is a summary of the pros and cons of each CI component:

At wix, we chose google cloud products in all three sub-areas, and are working closely with Google on improving the experience with each one.

The previous post in the series was on how to choose the right build unit granularity

The next post in the serires is about how to decide on the best local development setup for mac, linux and windows.

Thank you for reading!

Please also share on Facebook and Twitter. If you’d like to get updates, follow me on Twitter and Medium.

You can also visit my website, where you will find my previous blog posts, talks I gave in conferences and open-source projects I’m involved with.

If anything is unclear or you want to point out something, please comment down below.