Publishing native artifacts from a Kotlin Multiplatform project

Using Kotlin Multiplatform technology to produce native CocoaPods and NPM packages

In my previous post, Kotlin Multiplatform at Glovo, I told the story of how we introduced the Kotlin Multiplatform (from now KMP) technology to solve our Tracking Events business use case.

Here, I’ll dig into our solution's technical details, the challenges we faced (and how we solved them), and the tooling we built to support it.

I encourage you to first gather a basic understanding of the fundamentals of Kotlin Multiplatform, targeting JVM, JS, and Native/iOS. We’ll explore a different use case not yet explored by the community (at least that I am aware of), hoping it will be very interesting and enriching reading to you.

It’s a bit extensive, but I believe the topic is worth it.

The scale restriction

Assuming you have a bit of experience already with Kotlin Multiplatform, you may note (or may not) that it has to be “the core” of your project. And by “core” I mean the glue required to bind everything together.

Take a look at Kotlin’s official KMP example repo for instance. This project is a basic setup for a mobile app, and in there you can find the module for KMP, plus one per each technology: android and iOS.

I’m not going to dig deeper on Android’s setup because it’s pretty straightforward. On the other hand, if you look at the iOS’s wiring setup, you’ll notice that it’s done basically with an XCode’s build script running a Gradle task to build the project into a framework.

The same will apply if you eventually want to add a JavaScript target for a web version. The wiring will be a bit different but you basically need to have a dedicated module for your webapp.

A common structure of a KMP project

Any mature frontend probably has a big and complex codebase, CI/CD setups, and any kind of process around them. Probably, each one has its own repository as well.
Because of this reason, KMP may not be a good fit for organizations to adopt.

Forcing all your platform to join on a single repo (as shown in the previous screen), and changing the build-chain to integrate with Kotlin is probably the biggest barrier you will find if you try it.

That was our case at Glovo, but we managed to provide an alternative.

The native artifacts approach

What about if we manage to publish a CocoaPod and an NPM package? That was my thought when I faced the problem.

In theory, everything is there already. The KMP Plugin’s output provides a Framework for iOS and a JavaScript file for web. We just needed to build all the tooling to publish the packages in their native formats.

So we took the challenge and decided to do it in a “Gradle-friendly” approach by extending Gradle’s publishing plugin and Kotlin’s KMP one like this:

High-level of the tooling we have built

Gradle’s Publishing Plugin extensions

First, we’ve extended Gradle’s Publishing DSL with the new:

• CocoaPodsRepository and NPMRepository repository types (analogs to MavenArtifactRepository)
• PodPublication and PackagePublication publication types (analogs to MavenPublication)
• Podspec and PackageJson artifact descriptor types (analogs to MavenPom)

Gradle’s Maven Publish DSL example

This is our DSL for CocoaPods and NPM publication in Gradle:

Glovo’s CocoaPods Publish DSL example

Glovo’s NPM Publish DSL example

The plugins will add the necessary tasks to support publishing each repository in an isolated way. You don’t need to do anything outside Gradle’s build.

For instance, registering a new cocoaPods repository will create a SetupCocoaPodsRepositoryTask, which then sets up an isolated local tmp gems installation and executes pod repo add command on it.

Be aware this is highly coupled with Gradle’s internal API, and is likely to break across version changes.

Glovo’s KMP Native Library Plugin

Once we have set up Gradle to be able to handle CocoaPods and NPM artifact’s publishing it was time to start wiring everything together with JetBrains’s KMP plugin.

The NPM target

Well, not really a KMP new target, but an extension of KotlinJsTarget that also registers and links a PackagePublication. Applying this target ensures the project will produce an NPM library as part of its build process.

NPM target (extension of JS target) DSL example

The Pod target

Following the same approach, we also created a new pod target but in this case, it will rely on one or more iOS’s KotlinNativeTargets.

By default, this “virtual target” will create iosArm64 (for most modern iOS devices) and iosX64 (for iOS emulators), and you can optionally also include support for iosArm32 (older iOS devices).

The target will also register a PodPublication, so will be able to define and manipulate the artifact’s Podspec file. It will also register a FatFrameworkTask (provided by KMP’s plugin) whose output will be the main artifact of the publication and will merge all intermediate frameworks produced by each KotlinNativeTargets link task.

So the output of this target will be a fully functional multi-architectural XCode Framework, published as a CocoaPod.

Pod target (extension of iOS targets) DSL example

The Standard Layout concept

And finally, built upon all this tooling, the concept of a “standard layout” of a native library arises. It basically means that each KMP Gradle module may apply for android, ios, mobile (android+ios), web, or backend; following this sourceSet’s structure:

Hierarchy of `main` sourceSets, in our standard library layout convention

With its test counterparts:

Hierarchy of `test` sourceSets, in our standard library layout convention

And of course, our plugin adds a dedicated DSL for your Gradle build scripts:

Glovo’s Standard Library Layout DSL example

Designing to be as frictionless as possible

Given the reality of Glovo as a company with many cross-dependencies between teams, having as little impact as possible in consumer repos was crucial to the success and adoption of our library.

For that reason, we decided to keep the scope at the minimum possible to achieve our main goal: the library will just be responsible for mapping logical events (represented as classes) into raw JSON data (a merge from the event itself, the device and the user session). We won’t go into any complex implementation like persistence, http requests, etc.

This is how our main entry point looks like:

Basically, each consumer app has to provide an instance conforming to the AnalyticsDelegate interface. Every time an event is sent through the track method, it’s going to be serialized as raw data and then a callback function from the delegate will be invoked with it. Then, the app sends this raw data to our tracking endpoint.

The `event` callback method from the `AnalyticsDelegate` interface

If you have a trained eye, the previous screenshots revealed the first two frictions we had with the web and iOS team:

JS function mangling

Unless you give an explicit name with @JsName, functions with parameters will be mangled (adding a suffix) when compiled to JS. That’s because JS is not a typed language. So any parameterized function is a potential clash when overloaded (now or in future compilations of your code). I remember seeing some issues reported around this and proposing alternative approaches. I personally think this is fine, and it’s one of the tradeoffs you need to accept.

Example of Kotlin stdlib’s `mapOf` function mangling once compiled into JS

Lack of Thread Safe Mutable Collections in Native targets

The second is the threadSafeMutableMap(), which is actually an expect/actual function. For JVM targets, it will just use a ConcurrentHashMap. For web targets (remember it’s single-threaded) just a regular HashMap will do.

The problem was iOS, Kotlin Native to be precise, and its concurrency memory model that enforces immutability to achieve thread-safety. We didn’t want to add extra complexity to the project, by introducing coroutines (not a stable technology at that moment) for instance.

My first attempt to overcome this was to use Stately, from Touchlab (we actually had a podcast with them about our project, with my colleague Zeyad).

But at that time, they were not targeting JS (now they do).
So, at the end, we decided to have our own class implementing the interface, the actual value to be held by kotlin.native.concurrent.AtomicReference, and each mutable operation will try compareAndSet until it succeeds. Not perfect, but it works.

The Kotlin Native Memory model

At some point when you design a KMP solution, you will face this challenge.

Basically, the shared XOR mutable memory model in Kotlin Native enforces any mutable object to be accessed from the thread that has instantiated it. An exception will be thrown if this rule is violated.

To relax this limitation, you need to “freeze” the object. This means the object itself and all its references have to be “frozen” or immutable.

We wanted our KMP solution to be as simple as possible, so introducing coroutines or any complex solution was not an option. As mentioned in the previous topic, we decided to use our own mutable thread-safe collections. This decision has a price to pay in our commonMain logic.

There are a few ways to make an object “frozen” in the Kotlin Native framework. We decided to use the stdlib function kotlin.native.concurrent.freeze. The problem is that the function is only available in Native targets, and we were constructing objects in our common sourceSet.

To overcome this, we had to introduce an expect val <T> T.frozen: T function (other targets will just return this) and make our common code aware of this as you can see here:

Kotlin Native’s `frozen` aware code in `commonMain` sourceSet

KMP was not designed for this

The more I dig deeper into the framework, face and solve issues, the more I realize this was never considered as a possible use case of the KMP technology.

One of the biggest challenges we faced was around transitive dependencies, most precisely depending on other MPP modules.

Each target has its own problems with it which I’ll try to summarize here.

Our analytics solution has a base module with all the shared code, and then we have dedicated modules with specific events targeting different domains of our company. For instance, customer and courier modules define their own independent events, but they share the foundation logic depending on the base module.

For JVM build, everything works just fine through Maven support. But on iOS and web it’s another story.

iOS transitive module dependencies issues

iOS (native) builds don’t manage the concept of dependencies, they just create a binary bundle instead (a framework) and expose some symbols through Objective-C Header files.

Luckily there is a little known API to include any dependency into that Header file.

So we took the opinionated decision of exporting the whole apiConfiguration of the modules and that was it, all base module classes were now visible on XCode.

JavaScript transitive module dependencies issues

Similarly, in the JS approach, each KMP module gets compiled and translated to the project’s root as a local node module; which will later be run as a standalone app. So you will have a dedicated .js file per module and dependency resolution will be done through require or import.

One problem we observed when we published our first alpha NPM package was that the default .js contained only the classes defined in the module itself, but those from the base one were missing. Actually, we hit a runtime error trying to resolve an unknown base module (through the require) function.

To overcome this (again, an opinionated decision), we used a (not well known) feature of Node, where you can have a node_modules folder inside your NPM package. We just converted our regular NPM package into a bundled one. It required a bit of effort, but it works transparently.

Content of the `npm` package built by our tooling

The .bundle.js file is actually the package’s main entry point, which resolves and merges the exports of all the other packages.

I’m pretty sure this was changed in Kotlin 1.5 with the new IR compiler and now it’s producing a single .js bundle. I still haven’t had time to fully check it as we have other blocking issues preventing the migration (addressed at the end of this document in the “final thoughts” section).

Lessons learned

• Kotlin Native’s shared XOR immutable memory model makes multiplatform development much more complex and introduces an extra barrier to the adoption of the technology into the iOS world. I kind of remember seeing KEEP for relaxing this limitation (maybe with an opt-in feature) and just to provide a normal (or if you like it, “legacy”) approach to concurrency and threading.
  It seems this is going to be addressed soon:
  https://blog.jetbrains.com/kotlin/2021/05/kotlin-native-memory-management-update/
• Kotlin 1.5 has introduced some breaking changes on its JS/IR compiler, preventing interfaces and enums from being exported.
• KMP still has much room for improvements regarding interpolarity with other platforms than Java.

Conclusions

As I mentioned from the very beginning, this was for me just a research and exploration experimental project (I would use more “non-ready for production” words in here) that suddenly faced the opportunity to have a real use case to test.

I think we can consider it stable enough now, but very difficult to maintain. From a professional perspective, I’ve decided to stop scaling the technology and not introduce any new multiplatform libraries.

I’d love to see other companies continuing this research further and who knows, maybe even get official support from JetBrains and Gradle for it.

Having this as an out-of-the-box feature will definitely contribute to adopting the technology incrementally for brownfield native projects, as was our use case.