Recently, I managed to deploy several OSS artifacts built with Bazel to a Maven Central repository. As this process is quite complicated and involves many steps, I’ve decided to write a detailed step-by-step guide. I sure would have liked to have had one before I got started.
I work at Wix, as part of the data-streams team, that is in charge of all our Kafka infrastructure. Recently we started to open source our tools and libraries, starting with Greyhound — a Scala/Java High-level rich SDK for Apache Kafka.
Greyhound is built with Bazel — an open source build tool designed for speed and accuracy with truly incremental builds and a high degree of test parallelism.
In order to publish Greyhound’s logical artifacts to maven central using Bazel, we needed to perform some preparations.
First, Bazel does not publish build output jars in a local/remote maven repository by default. A separate publishing ruleset is needed. Thankfully, graknlabs has created and open-sourced such a ruleset.
Second, with Bazel it is encouraged to have smaller build unit targets at java package level — it doesn’t make sense to publish each individual package as a complete module to Maven central, such that an additional target that aggregates all the relevant packages has to be created for each logical maven module.
The guide below provides all required steps to deploy to Maven central including all Bazel related setup.
Step 1: Create a Sonatype JIRA Ticket
Once you are logged in, create a new issue.
You need to do this in order to allow your Sonatype user credentials to publish your OSS project’s Maven modules to Maven central.
You need to specify group id, project url on github, etc…
It will take some time for the administrator to get back to you — that’s why this is step 1.
Step 2: create and distribute GPG key
In order to be able to successfully publish your artifacts, they are required to be signed using GPG keys.
You need to make sure to create a set of private/public keypair on the machine you are going to publish the artifacts from.
On a Mac, the easiest way is to install GPG Keychain, and follow the instructions on how to create a new keypair.
Alternatively, for linux machines, you can use GnuPG.
Don’t forget to also publish the public key to a key server, so that your signed artifacts would be verifiable.
You can verify that the keypair is available by running
Step 3: Setup Bazel configuration
1. add a dependency on bazel-distribution repo from graknlabs
2. for each artifact you want to publish to maven central, create an aggregate bazel target that includes sources for all finer-grained bazel targets (if you have bazel targets with coarse-grained granularity you can skip this step)
A few notes:
- This is a
scala_library, but this can work for
- All the labels in srcs are actually
filegroupwith appropriate visibility settings that allow out-of-package dependency:
- This target is tagged as
"manual", so it won’t be needlessly built when running
bazel build //core/…as any source change would cause the whole source set to be rebuilt for this specific target.
- There is a special
maven_coordinatestag that helps the
assemble_mavenrule to fill-in details in the pom file.
- You also have to add a
maven_coordinatestag to the 3rd party dependencies (such as
depsattribute of the example above) so that they will appear as dependencies in the pom file:
assemble_maven target for each artifact you want to publish. It will create all the necessary files for your artifact, including main jar, source jar and pom file. Enter all project details for the pom file.
- For the
targetattribute you should put the label for the Bazel target (
java_library) you created in the previous step with all the relevant sources.
- Make sure the
project_descriptionare unique for each of these targets/artifacts
VERSIONfile just contains the SEMVER, e.g. 1.0.0
- choose from either
deploy_maven for each artifact you want to publish.
deployment.properties file should contain the url for the sonatype stating area:
For a complete working solution you can review our greyhound repo.
5. Build the
assemble_maven target and review the generated pom file and jar file in the bazel-bin directory. Change the targets configuration as needed.
Step 4: publish via Bazel to Sonatype staging area
Once you get the ok from the sonatype administrator in the JIRA ticket you opened in step 1, you can try and publish the artifacts to the Sonatype staging area.
You need to have your Sonatype account credentials (that you created in step 1) available as environment variables for successful execution of the
make sure that you escape any special characters found in the password.
For each artifact, execute:
bazel run //path/to/artifact:deploy-maven -- release --gpg
- Make sure that you didn’t skip step 2 and the GPG keypair is available on your machine by running
- Make sure you put the version you want in the VERSION file that assemble_maven depends on
If the run is successful you will find the artifacts in https://oss.sonatype.org/#stagingRepositories
Check the repo checkbox and in the bottom part, go to the content part and review the artifacts content.
You can click on each item and review its content. I recommend that you review the pom file content, download the jars and verify their content using
jar tf path/to/jar/file.jar.
Step 5: Auto-evaluation of the published artifacts content
When you are satisfied with the content, the next step is for Sonatype to automatically verify that the content is valid.
Click on Close repo.
The evaluation process will start. This can take a few minutes.
It includes verifying that the pom file is in order and that the GPG signature is valid.
You can review the evaluation results in the Activity tab.
When the evaluation is completed successfully, the repository will close.
Step 6: Release the closed staged repo to Central
Click on Release in order to start the sync process of the artifacts to Maven central itself.
This can take several hours, up to a day.
Once the sync is completed, you will be able to find the artifacts in
Congratulations! You have successfully published your artifacts from Bazel to Maven Central.
Here are links to my blog series on how to successfully migrate to Bazel from Maven or Gradle:
- How to choose the right build unit granularity
- How to Decide on CI Server and remote execution / caching
- How to optimize local dev experience on MacOs, Windows and Linux
- Can Bazel work with Manyrepo?
- How to avoid Jar dependency hell with Bazel
Also a link to an article about Exodus — our automatic migrator tool from Maven to Bazel.
Thank you for reading!
If you’d like to get updates on my experiences with Bazel, Kafka and OSS projects please follow me on Twitter and Medium.
You can also visit my website, where you will find my previous blog posts, talks I gave in conferences and open-source projects I’m involved with.
If anything is unclear or you want to point out something, please comment down below.