sbt-sonatype: Blazingly Fast Release to Sonatype
If you are a maintainer of Scala OSS projects and have never tried sbt-sonatype 3.6 (or higher), your release process can be much faster by using the bundle upload feature of sbt-sonatype:
The bundle upload enables uploading multiple artifact files of your project with a single HTTP request to Sonatype, a central repository for JVM-based OSS projects. With the bundle upload, for example, more than 2000+ files of Airframe project can be uploaded in a minute, even though it has 20+ modules and cross-built binaries for Scala 2.11, 2.12, 2.13, and Scala.js.
Traditionally, Scala developers have been using publishSigned task of sbt, which uploads artifact files one by one. Probably because of the inefficiency of ivy2 used inside sbt, uploading thousands of files to Sonatye could take several hours. This has been painful, especially if you are maintaining cross-build Scala projects with multiple modules.
After upgrading to the latest sbt-sonatype, for example, sttp, a Scala http client library, can be released in 34 minutes instead of 3 hours!
Alex, the creator of coursier, was also surprised with the performance of the bundle upload:
His another project, almond, a Scala kernel for Jupyter Notebook, no longer needs 9 days (!!!) for a single release:
sbt-coursier library, which is integrated into sbt 1.3.x, makes reading artifacts from Sonatype (a.k.a. Maven Central) significantly faster. Now both reading and uploading artifacts can be super-fast with the latest sbt and sbt-sonatype. Our life, as OSS developers, is now much easier and fun.
If you already have an account at Sonatype OSS repository, the following publishTo setting is necessary to create a local staging folder before uploading artifacts as a bundle:
publishTo := sonatypePublishToBundle.valueWith this setting, publishSigned task will create a local staging folder like target/sonatype-staging/(version) and collect all of the artifact files of your projects, such as .jar, .asc (GPG signature), .sha1 (checksum), etc. into this folder. Even if you have multi-module projects, creating a single staging folder is sufficient as we can upload them as a whole.
Then, you need to run publishSigned as usual, and sonatypeBundleRelease command to upload the bundle and verify your project’s Maven central requirements at Sonatype:
> ; publishSigned; sonatypeBundleReleaseThat’s it. It’s surprisingly easy, isn’t it?
Scala community, however, has been struggled to overcome the slow upload performance to Sonatype for years. Especially since 2018, it’s been quite painful to release projects to Sonatype because the API performance becomes significantly slower than before:
As the creator of sbt-sonatype, I should have known the presence of the bundle API, but the information is quite limited as you can see in the screenshot of this REST API documentation of Nexus, which is the service for hosting Sonatype OSS repository:
Yes. I knew this API does exist since 2015 when I created sbt-sonatype. But I almost had no idea how to use this API. I also didn’t know what are bundles. Even for using the other API endpoints, I somehow deciphered how to use them with a try-and-error approach when implementing sbt-sonatype.
Aha! moment came later after checking the source code of some Nexus libraries. Actually, Pavel already mentioned a workaround that uses this bundle API in sbt and sbt-sonatype GitHub issues:
His workaround script was using Ant, and I guess almost nobody in Scala community has a serious experience of using Ant, so connecting dots should have been a bit difficult. Fortunately, I was using Ant more than 10 years ago and I knew it is just a Java program, so I thought it’s worth checking the actual implementation, and found the solution was that easy.
Next time, I’m not sure what it will be, but I hope we can connect dots more easily without waiting for years.
Automating Release on CI
If you want to perform the release process on CI, using sbt-ci-release, or reading its source code (it’s actually a quite small sbt plugin) will be helpful:
To automate release, basically, you need to encode your GPG private key and its passphrase as environment variables in CI, and decode them during the release build. Then you can apply GPG signature to artifact files at publishSigned task. sbt-ci-release is using the latest sbt-sonatype, so the bundle upload is also used internally. Using sbt-dynver to automate project versioning is a key to avoid manual versioning process.
Now it’s also possible to use GitHub actions to release your projects:
You can find more general tips for maintaining Scala OSS projects from here:
Next Step
I’m now proposing adding publishStaging task to sbt to make it easier to create a local staging folder. As we already have publishLocal, publishM2 tasks in sbt, I think adding publishStaging task should be straightforward.
Thank you for reading. Enjoy OSS programming with Scala!