Continuous delivery for your Kotlin Multiplatform library
As part of the Bumble family— the parent company operating Badoo and Bumble apps — one of my main projects was involved in the team that created Reaktive library — Reaktive Extensions on pure Kotlin.
In the case of Kotlin Multiplatform I discovered that continuous integration and continuous delivery require additional configuration. You need to have multiple virtual machines with different operating systems in order to build a library. In this article, I’ll be showing you how to configure continuous delivery for your Kotlin Multiplatform library.
Continuous integration and continuous delivery for open-source libraries
Continuous integration and continuous delivery have been part of the open-source communities for a long time as they offer a number of useful services. Many of them provide their services for open-source projects completely free of charge: Travis CI, JitPack, CircleCI, Microsoft Azure Pipelines and also GitHub Actions, which launched recently.
For Badoo open-source projects for Android we use Travis CI for continuous integration and JitPack for continuous delivery.
Following the implementation of iOS support in our multi-platform library, I discovered that we couldn’t build the library using JitPack, because it doesn’t provide virtual machines on macOS (iOS can only be built on macOS).
So, a more familiar Bintray was chosen for further publication of the library. It allows published artifacts to be more finely tuned, unlike JitPack, which simply took all of the results of the publishToMavenLocal task.
The Gradle Bintray Plugin is recommended for publishing, and I later configured to suit our needs. To build the project, I continued to use Travis CI for several reasons: firstly, I was already familiar with it and I had used it for nearly all of my pet projects; secondly, it provides virtual machines on macOS, which is necessary for building for iOS.
Parallel building of multiplatform library
If you look at the Kotlin documentation in detail, you will find a section on publishing multiplatform libraries.
Kotlin Multiplatform developers are aware of the multiplatform building issues (not everything can be built on all operating systems) and offer the option of building the library separately on different operating systems.
kotlin {
jvm()
js()
mingwX64()
linuxX64()
// Note that the Kotlin metadata is here, too.
// The mingwx64() target is automatically skipped as incompatible in Linux builds.
configure([targets["metadata"], jvm(), js()]) {
mavenPublication { targetPublication ->
tasks.withType(AbstractPublishToMaven)
.matching { it.publication == targetPublication }
.all { onlyIf { findProperty("isLinux") == "true" } }
}
}
}As the code above shows, depending on the ‘isLinux’ property passed to Gradle, we enable the publishing of certain targets. By targets I mean the assembly for a specific platform. On Windows, only the Windows-target will be assembled, while on other operating systems metadata and other targets will.
An excellent and concise solution that only works for publishToMavenLocal or publish from the maven-publish plugin, which is not suitable for us because of the use of Gradle Bintray Plugin.
I decided to use the environment variable for selecting the target, as this code was previously written in Groovy, was in a separate Groovy Gradle script, and access to the environment variables is from a static scope.
enum class Target {
ALL,
COMMON,
IOS,
META; val common: Boolean
@JvmName("isCommon")
get() = this == ALL || this == COMMON val ios: Boolean
@JvmName("isIos")
get() = this == ALL || this == IOS val meta: Boolean
@JvmName("isMeta")
get() = this == ALL || this == META companion object {
@JvmStatic
fun currentTarget(): Target {
val value = System.getProperty("MP_TARGET")
return values().find { it.name.equals(value, ignoreCase = true) } ?: ALL
}
}
}
As part of our project, I have identified four target groups:
- ALL — all targets are set up and assembled and used for development and as a default.
- COMMON — only Linux-compatible targets are set up and assembled. In our case, this is JavaScript, JVM, Android JVM, Linux x64 and Linux ARM x32.
- IOS — only iOS-targets are set up and assembled; used for assembly on MacOS.
- META — all targets are set up, but only the module with meta information for Gradle Metadata is assembled.
With these target groups, we can parallelise the assembly of the project on three different virtual machines (COMMON — Linux, IOS — macOS, META — Linux).
Currently, it is possible to assemble everything on macOS, but my solution has two advantages. Firstly, if we decide to implement Windows support, all we need to do is add a new target group and a new virtual machine on Windows. Secondly, there is no need to spend virtual machine resources on macOS for items that can be assembled on Linux. CPU time on these virtual machines is usually twice as expensive.
Gradle Metadata
What is Gradle Metadata and what is it for?
Currently, Maven uses POM (Project Object Model) to resolve dependencies.
<?xml version="1.0" encoding="UTF-8"?>
<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<modelVersion>4.0.0</modelVersion>
<groupId>com.jakewharton.rxbinding2</groupId>
<artifactId>rxbinding-leanback-v17-kotlin</artifactId>
<version>2.2.0</version>
<packaging>aar</packaging>
<name>RxBinding Kotlin (leanback-v17)</name>
<description>RxJava binding APIs for Android's UI widgets.</description>
<url>https://github.com/JakeWharton/RxBinding/</url>
<licenses>
<license>
<name>The Apache Software License, Version 2.0</name>
<url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
<distribution>repo</distribution>
</license>
</licenses>
<developers>
<developer>
<id>jakewharton</id>
<name>Jake Wharton</name>
</developer>
</developers>
<scm>
<connection>scm:git:git://github.com/JakeWharton/RxBinding.git</connection>
<developerConnection>scm:git:ssh://git@github.com/JakeWharton/RxBinding.git</developerConnection>
<url>https://github.com/JakeWharton/RxBinding/</url>
</scm>
<dependencies>
<dependency>
<groupId>com.android.support</groupId>
<artifactId>support-annotations</artifactId>
<version>28.0.0</version>
<scope>compile</scope>
</dependency>
</dependencies>
</project>The POM file contains information about the library version, its creator and the required dependencies.
What if we want to have two library versions for different JDKs? For example, there are two versions of kotlin-stdlib: kotlin-stdlib-jdk8 and kotlin-stdlib-jdk7. Users need to connect to the required version themselves.
When updating the JDK version, it is very easy to forget the external dependencies. Gradle Metadata was created in order to solve this problem; it allows you to add additional conditions for a particular library.
One of the supported Gradle Metadata attributes is org.gradle.jvm.version, which specifies the JDK version. Therefore, a simplified metadata file for kotlin-stdlib might look like this:
{
"formatVersion": "1.0",
"component": {
"group": "org.jetbrains.kotlin",
"module": "kotlin-stdlib",
"version": "1.3.0"
},
"variants": [
{
"name": "apiElements",
"attributes": {
"org.gradle.jvm.version": 8
},
"available-at": {
"url": "../../kotlin-stdlib-jdk8/1.3.0/kotlin-stdlib-jdk8.module",
"group": "org.jetbrains.kotlin",
"module": "kotlin-stdlib-jdk8",
"version": "1.3.0"
}
},
{
"name": "apiElements",
"attributes": {
"org.gradle.jvm.version": 7
},
"available-at": {
"url": "../../kotlin-stdlib-jdk7/1.3.0/kotlin-stdlib-jdk7.module",
"group": "org.jetbrains.kotlin",
"module": "kotlin-stdlib-jdk7",
"version": "1.3.0"
}
}
]
}In our case, reaktive-1.0.0-rc1.module in simplified form looks like this:
{
"formatVersion": "1.0",
"component": {
"group": "com.badoo.reaktive",
"module": "reaktive",
"version": "1.0.0-rc1",
"attributes": {
"org.gradle.status": "release"
}
},
"createdBy": {
"gradle": {
"version": "5.4.1",
"buildId": "tv44qntk2zhitm23bbnqdngjam"
}
},
"variants": [
{
"name": "android-releaseRuntimeElements",
"attributes": {
"com.android.build.api.attributes.BuildTypeAttr": "release",
"com.android.build.api.attributes.VariantAttr": "release",
"org.gradle.usage": "java-runtime",
"org.jetbrains.kotlin.platform.type": "androidJvm"
},
"available-at": {
"url": "../../reaktive-android/1.0.0-rc1/reaktive-android-1.0.0-rc1.module",
"group": "com.badoo.reaktive",
"module": "reaktive-android",
"version": "1.0.0-rc1"
}
},
{
"name": "ios64-api",
"attributes": {
"org.gradle.usage": "kotlin-api",
"org.jetbrains.kotlin.native.target": "ios_arm64",
"org.jetbrains.kotlin.platform.type": "native"
},
"available-at": {
"url": "../../reaktive-ios64/1.0.0-rc1/reaktive-ios64-1.0.0-rc1.module",
"group": "com.badoo.reaktive",
"module": "reaktive-ios64",
"version": "1.0.0-rc1"
}
},
{
"name": "linuxX64-api",
"attributes": {
"org.gradle.usage": "kotlin-api",
"org.jetbrains.kotlin.native.target": "linux_x64",
"org.jetbrains.kotlin.platform.type": "native"
},
"available-at": {
"url": "../../reaktive-linuxx64/1.0.0-rc1/reaktive-linuxx64-1.0.0-rc1.module",
"group": "com.badoo.reaktive",
"module": "reaktive-linuxx64",
"version": "1.0.0-rc1"
}
},
]
}Thanks to the org.jetbrains.kotlin attributes, Gradle knows when a particular dependency needs to be pulled into the required source set.
Metadata can be enabled using:
enableFeaturePreview(“GRADLE_METADATA”)Detailed information can be found in this documentation.
Publishing configuration
Once we have dealt with the targets and assembly parallelisation, we need to configure exactly what we are going to publish and how we’ll do it.
For publishing, we use Gradle Bintray Plugin, so first of all, we will refer to its README and configure information about our repository and the credentials for publishing.
The entire configuration will be performed in our own plugin in the buildSrc folder.
There are a number of advantages of using buildSrc. For example, autocomplete works in nearly all cases (except for Kotlin scripts where it doesn’t always work and often requires an apply dependencies call), classes from it can be reused and accessed from Groovy and Kotlin scripts. You can see a usage example here buildSrc from the latest Google I/O (Gradle section).
private fun setupBintrayPublishingInformation(target: Project) {
// We apply Bintray Plugin to the project
target.plugins.apply(BintrayPlugin::class)
// And we configure it
target.extensions.getByType(BintrayExtension::class).apply {
user = target.findProperty("bintray_user")?.toString()
key = target.findProperty("bintray_key")?.toString()
pkg.apply {
repo = "maven"
name = "reaktive"
userOrg = "badoo"
vcsUrl = "https://github.com/badoo/Reaktive.git"
setLicenses("Apache-2.0")
version.name = target.property("reaktive_version")?.toString()
}
}
}I use three dynamic project properties: bintray_user and bintray_key, which can be retrieved from the personal profile settings on Bintray, and the reaktive_version, which is set in the build.gradle root file.
For each target, Kotlin Multiplatform Plugin creates MavenPublication, which is available in PublishingExtension.
Using the example code from the Kotlin documentation, which I mentioned above, we can create this configuration:
private fun createConfigurationMap(): Map<String, Boolean> {
val mppTarget = Target.currentTarget()
return mapOf(
"kotlinMultiplatform" to mppTarget.meta,
KotlinMultiplatformPlugin.METADATA_TARGET_NAME to mppTarget.meta,
"jvm" to mppTarget.common,
"js" to mppTarget.common,
"androidDebug" to mppTarget.common,
"androidRelease" to mppTarget.common,
"linuxX64" to mppTarget.common,
"linuxArm32Hfp" to mppTarget.common,
"iosArm32" to mppTarget.ios,
"iosArm64" to mppTarget.ios,
"iosX64" to mppTarget.ios
)
}In this simple map, we illustrate which publications should be released on a specific virtual machine. The name of the publication is the name of the target. This configuration is completely consistent with the target groups description which I gave above.
private fun setupBintrayPublishing(
target: Project,
taskConfigurationMap: Map<String, Boolean>
) {
target.tasks.named(BintrayUploadTask.getTASK_NAME(), BintrayUploadTask::class) {
doFirst {
// Configuration here
}
}
}Anyone who begins working with the Bintray plugin quickly realises that the repository has been gathering dust for a while (the last update was about six months ago), and that all of the problems can be solved with all sorts of hacks and temporary solutions in the Issues tab. Support for a technology as new as Gradle Metadata has not been set up, but with an issue, you can find a solution, which is the one that we use.
val publishing = project.extensions.getByType(PublishingExtension::class)
publishing.publications
.filterIsInstance<MavenPublication>()
.forEach { publication ->
val moduleFile = project.buildDir.resolve("publications/${publication.name}/module.json")
if (moduleFile.exists()) {
publication.artifact(object : FileBasedMavenArtifact(moduleFile) {
override fun getDefaultExtension() = "module"
})
}
}With this code, we add to the list of artifacts for publishing the file module.json, which enables Gradle Metadata to work.
However, this is not the last of our problems. When you try to run bintrayPublish, nothing happens.
In the case of regular Java and Kotlin libraries, Bintray automatically takes the available publications and publishes them. However, in the case of Kotlin Multiplatform, the plugin simply crashes with an error. Speaking of which, there is an issue on GitHub for this, too. And we will use the solution from there again, only filtering the publications we need.
val publications = publishing.publications
.filterIsInstance<MavenPublication>()
.filter {
val res = taskConfigurationMap[it.name] == true
logger.warn("Artifact '${it.groupId}:${it.artifactId}:${it.version}' from publication '${it.name}' should be published: $res")
res
}
.map {
logger.warn("Uploading artifact '${it.groupId}:${it.artifactId}:${it.version}' from publication '${it.name}'")
it.name
}
.toTypedArray()
setPublications(*publications)But this code doesn’t work either!
This is because bintrayUpload doesn’t have a task in the dependencies able to build the project and create the files needed for publication. The most obvious solution would be to set publishToMavenLocal as a bintrayUpload dependency, but it’s not as simple as that.
When assembling metadata, we set up all of the targets to the project. This means that publishToMavenLocal will result in all of the targets being compiled, as the dependencies for this task are publishToMavenLocalAndroidDebug, publishToMavenLocalAndroiRelase, publishToMavenLocalJvm, etc.
We will therefore create a separate proxy task, and we will only put the publishToMavenLocalX tasks that we need in the dependency, and we will put the task itself as a dependency of bintrayPublish.
private fun setupLocalPublishing(
target: Project,
taskConfigurationMap: Map<String, Boolean>
) {
target.project.tasks.withType(AbstractPublishToMaven::class).configureEach {
val configuration = publication?.name ?: run {
// The Android-plugin does not immediately set the publication of the PublishToMaven task, which is why we use the heuristic method to find its name
val configuration = taskConfigurationMap.keys.find { name.contains(it, ignoreCase = true) }
logger.warn("Found $configuration for $name")
configuration
}
// We enable or disable the task depending on the current configuration
enabled = taskConfigurationMap[configuration] == true
}
}private fun createFilteredPublishToMavenLocalTask(target: Project) {
// We create a proxy task and set it to depend only on the publishToMavenLocal tasks
target.tasks.register(TASK_FILTERED_PUBLISH_TO_MAVEN_LOCAL) {
dependsOn(project.tasks.matching { it is AbstractPublishToMaven && it.enabled })
}
}
All that is left to do is to assemble all code together and apply the resulting plugin to the project in which publication is required.
abstract class PublishPlugin : Plugin<Project> {
override fun apply(target: Project) {
val taskConfigurationMap = createConfigurationMap()
createFilteredPublishToMavenLocalTask(target)
setupLocalPublishing(target, taskConfigurationMap)
setupBintrayPublishingInformation(target)
setupBintrayPublishing(target, taskConfigurationMap)
}apply plugin: PublishPlugin
You can find the complete PublishPlugin code in our repository here.
Travis CI configuration
The hardest part is over. Travis CI still needs to be configured so that it parallelises the assembly and publishes the artifacts in Bintray when a new version is released.
When a new version is released, we will create a tag on the commit.
# We use the matrix build (parallel execution)
matrix:
include:
# On Linux, Android and Chrome to assemble JS, JVM, Android JVM and Linux targets
- os: linux
dist: trusty
addons:
chrome: stable
language: android
android:
components:
- build-tools-28.0.3
- android-28
# We use MP_TARGET in order to set the required target group for assembly
env: MP_TARGET=COMMON
# We can skip the install step — Gradle will bring up all the dependencies itself
install: true
# When assembling in JVM, we also build a compatibility library with RxJava2
script: ./gradlew reaktive:check reaktive-test:check rxjava2-interop:check -DMP_TARGET=$MP_TARGET
# On macOS for assembling iOS-targets
- os: osx
osx_image: xcode10.2
language: java
env: MP_TARGET=IOS
install: true
script: ./gradlew reaktive:check reaktive-test:check -DMP_TARGET=$MP_TARGET
# On Linux to assemble metadata
- os: linux
language: android
android:
components:
- build-tools-28.0.3
- android-28
env: MP_TARGET=META
# Metadata assembly does not require any verification
install: true
script: true
# Recommended Gradle caching settings (to avoid loading all dependencies between builds on the same branch every time)
before_cache:
- rm -f $HOME/.gradle/caches/modules-2/modules-2.lock
- rm -fr $HOME/.gradle/caches/*/plugin-resolution/
cache:
directories:
- $HOME/.gradle/caches/
- $HOME/.gradle/wrapper/
# We start publishing artifacts in Bintray when creating a new tag, this block will be run on each virtual machine from the matrix
deploy:
skip_cleanup: true
provider: script
script: ./gradlew bintrayUpload -DMP_TARGET=$MP_TARGET -Pbintray_user=$BINTRAY_USER -Pbintray_key=$BINTRAY_KEY
on:
tags: trueIf for some reason the assembly on one of the virtual machines does not work properly, the metadata and other targets will still be uploaded to the Bintray server. That is why we do not add a block with automatic library release on Bintray through their API.
When the version is released, you need to make sure that everything is in order, and simply click on the button to publish a new version on the site, as all the artifacts are already uploaded.
Conclusion
We have used this process to set up continuous integration and continuous delivery for our Kotlin Multiplatform project.
By parallelising the tasks of assembling, running tests and publishing artifacts, we have made effective use of the free resources available to us.
And if you use Linux, you no longer need to ask someone using macOS to publish the library every time a new version is released.
I hope that after this article is published, more developers will start using this approach to automate routine actions for their projects.
Thanks for reading!
