Migrating to Bazel from Maven or Gradle? Part 3 — How to optimize local dev experience

Bazel’s abilities vary considerably on different operating systems, with Linux being superior to windows/MacOs.

In the following sections, I will describe how crucial bazel features for fast and reproducible builds are implemented on the different OSs.
I will also describe the trade-offs of a virtualization solution for local development with Bazel.

Reproducible Builds with Sandboxing

One of the most important features of Bazel is sandboxing.
Sandboxing allows for hermetic builds and safe caching of build outputs. It restricts access to files that were not explicitly defined in the build configuration. It also restricts access to the outside network.

Bazel implement sandboxing in Linux using User Namespaces that offers kernel support for file system and network isolation, and on macs using a feature called “SeatBelt” (sandbox-exec).

There is currently no windows support for Sandboxing.

There is an experimental sandboxing feature called sandboxfs that offers platform independent sandboxing that utilizes the FUSE file system. Preliminary performance results show faster performance for certain use cases, but it is still very early days for sandboxfs.

Parallel Tests

Another important feature of bazel is that it runs test actions in a highly parallel fashion.

Thus your integration tests that contain network requests must choose random ports or otherwise have to run in isolation from one another.
Bazel solves the isolation requirement using the same sandboxing techniques mentioned above.

While working well on Linux, port isolation is not supported on macOS. Even worse — binding to localhost is not supported on mac sandbox.

No matter which platform you’re running on, expect to have to adapt your code for sandboxing (e.g. stop using InetAddress.getLocalHost).

Remote Caching

A very cool feature bazel has is the possibility to locally utilize a remote cache (populated by servers or other developers). This way you never have to run a clean build, even on a brand new computer!

Unfortunately if your CI server are running on Linux agents, you will not be able to utilize the CI cache results on macs or windows, only on Linux machines. This is because bazel considers the platform as part of the hash function input for caching build targets.

Nevertheless you can still create another remote cache server for your local development environment. You need to decide whether to populate it using a dedicated CI server or developer machines or both.

There is a danger of cache poisoning in case of cache population from developer machines running bazel with default configurations. For instance a machine with a different JDK version can send incorrect builds results to the shared remote cache.

You can setup a specific jdk installation link using special bazel flags, but the best way to avoid surprises is to employ strict configuration management (like jamf for macs) on all dev machines.

Remote Execution

If you would like to employ remote execution, i.e. send build actions to a remote workers farm based on Linux, you have to be running locally on the same platform, in this case, Linux.

Even on Linux, running build actions remotely can have a considerable latency overhead that may outweigh the benefits of a larger degree of parallelization. Google remote execution service currently only supports Linux.

A new and experimental feature of Bazel called dynamic scheduling can help you enjoy both worlds by sending the actions to the remote execution service while simultaneously running the actions locally, It will cancel the slower action once the first one finishes.

IDEs

There is a bazel plugin for both intellij and Eclipse. It delegates the work to bazel by invoking it in command line, so these plugins are platform-independent.
But there are known debugging issues for Intellij bazel plugin for windows.

On the one hand, the delegation of work to bazel cli by the plugin insures correctness and a single source of truth. But on the other hand, it requires to call bazel build every time a developer wants to update Intellij knowledge of the code structure.

On macs, setting this sync to be automatic will sometimes stop the developer from being able to work in intellij for the duration of the build. The upcoming Bazel 0.24 which will utilize Darwin’s QoS service may resolve this issue.

Refactoring code in intellij will not get bazel’s BUILD files automatically fixed, which can make developers think twice before performing massive cross repo refactoring.

At Wix we are working hard on improving the ease of maintaining BUILD targets dependencies and hope to OSS most of the work soon.

Running Bazel on VMs/Docker containers

Running bazel inside a virtual linux environment (VM/docker) solves all of the sandboxing issues and thus provides reproducible builds, parallel tests and the ability to re-use your CI’s cache.

But these cool features could be negated by bad performance or data corruption.

The holy grail here is to have good performance when synchronizing bazel outputs back to your local machine where you are running the IDE.

Depending on your codebase, there could be potentially thousands of files that need to be synced back on each bazel build.

The intellij bazel plugin relies on these files in order to re-index and provide the great IDE functionality we are used to.

Potential file synching mechanisms:

• Docker mount point — performance is terrible
• Nfs mount — good performance but unreliable
• Smb mount — good performance, more reliable but still problematic
• Rsync — correct but performance depends on the amount of files

Recently Li Haoyi from Databricks wrote a blog post about their experience with Bazel, and in it they mention devbox, a remote vm that can run bazel. They claim to have superior file syncing capabilities, although it does not seem like their developers use the intellij plugin (it mentions running terminal commands over SSH)

Here is a summary of the pros and cons of each dev environment:

Setup at Wix

At Wix, most of our developers are using macs, so we investigated thoroughly the option of running bazel inside a linux docker container.
After experimenting with several different file syncing mechanisms, we came to realize that none of them give us the performance we require, especially as Intellij is universally used by our backend developers.

So we decided to have everyone run bazel natively, and have created a separate remote cache just for macs. We are actively working to make sure there is no chance of cache poisoning by the end user.
We are also actively working on improving the Intellij experience for Bazel by contributing to the Bazel Intellij plugin OSS.

The previous post in the series was on How to Decide on CI Server and remote execution / caching

The next post is about Bazel performance with many source control repositories.

Thank you for reading!

Please also share on Facebook and Twitter. If you’d like to get notified on the next installments, follow me on Twitter and Medium.

You can also visit my website, where you will find my previous blog posts, talks I gave in conferences and open-source projects I’m involved with.

If anything is unclear or you want to point out something, please comment down below.