This is the second (more technical) part of our story about putting what was possibly the first app on the Play Store with a significant Swift component. See here for part one.
As I described in the first part of this series, we had time and business pressure to release a flowkey Android app, with no proven technology path to achieve it. In summary, we decided to take our chances that Swift would become available to us on Android to run our performance-critical Pitch Detection code once Swift went open source (at a then-unknown point in time) without having to rewrite that code in C(++). This would accompany our hybrid largely HTML5-based app which had already proven itself performant and compatible running on Android up until then.
Preparing our Swift Code
In a way we were banking on a miracle, but as with all miracles that turn out for the best we had reason on our side: we had a good idea the plan was realistic due to insights learned from Romain Goyet’s post about how LLVM works (including a basic, but working example on Android). The timeframe also seemed doable thanks to Apple’s suggestion that Swift would be open sourced by the end of 2015. So we took the dive.
And the good news came quickly: within a week of deciding to give Swift a try on Android, swift.org went live along with Apple’s corresponding open source GitHub repository. Support for Linux was not 100% complete but for practical purposes ready ‘out of the box’. Although we still couldn’t compile Swift to Android immediately (which is a Linux system at its core), we very quickly got a good idea of what would be available to us in Swift and what wouldn’t. The biggest hurdles we saw would be in two areas:
- The Foundation framework: Math functions
- The Accelerate framework: Audio Filtering
Foundation-specific Math
We didn’t expect Foundation to be available to us from the start (actually we assumed it wouldn’t be available at all). We also wanted to avoid dancing around with conditional import statements at the top of every source file.
Instead, we decided to promote code readability and maintainability in our codebase (perhaps at the cost of potential bugs) by setting up a level of indirection. We made one file for each platform that would import the functions we needed and provide them at a Framework-internal access level to the rest of our code. So we’d compile FlowMath.swift for Darwin targets like iOS — importing Foundation; and FlowMathAndroid.swift for Android — here importing Glibc.
We also wanted to be able to just write, for example, sin(x) for both Float and Double types, regardless of platform. This worked in Foundation but not in Glibc (which is a C library, and by definition unaware of Swift’s function overloading). So writing our own wrapper here also meant for the most part we didn’t have to change the rest of our codebase at all. We also wrote our own versions of some higher-order functions like mean() and median() of floating point arrays, which had used some Apple-optimised functions on iOS.
Overall, this allowed us to share much more code between Android and iOS than we could have otherwise, and kept the implementation details clearly encapsulated in one place.
Audio Filtering
One of the most important aspects of what we were doing in our native Swift code was filtering audio (essentially EQing, for the not-so-audio-minded among us). We had been using a library called NVDSP for this purpose, which internally made use of a relatively obscure function in Apple’s Accelerate framework to run a recursive filter, vDSP_deq22():
From what I understood, the author of NVDSP, Bart Olsthoorn, had used a Google-translated version of a Japanese blog to understand how to use the function. The library had served us well up until then, but the thought was not exactly encouraging when we needed a deep enough understanding to implement the function ourselves by hand.
Luckily, Apple’s documentation does in fact reveal exactly what the function does beneath the surface (consult the friendly formula above). After some experimentation and frustration we were eventually able to write our own version of the filter function that produced the same output given a certain input (the necessary performance optimisations came much later of course).
Trying it out
So, we had a code base that would theoretically run on Android, but no compiler to generate the binary and no standard library to link it with. This was a major issue. Our entire four-person development team at flowkey has its roots firmly in web development meaning Swift itself was a learning curve, let alone compiler hacking and porting. I tried poking around in the Swift source code, but tackling the problem alone of trying to get the Swift standard library to compile for a new platform was one that I was ultimately entirely unqualified for.
Thankfully, an unassuming hacker and quiet hero by the name of Zhuowei Zhang (@zhuowei on GitHub) had also started to make progress and share it with the world (this is a squashed commit of the final outcome). I quickly expressed interest in the endeavour and began ‘helping’ wherever I could. To begin with, that involved little more than testing the latest progress @zhuowei had made on the SwiftAndroid repository, and reporting on how it worked with real code (@zhuowei had only limited experience with Swift at the time). In that way, it took only a few weeks to get all of Swift’s features working on an Android device.
But the work was far from over. Having access to a reliable compiler and a fully-fledged Swift standard library to make binaries for the Android command line was as exciting as it was promising, but it still left a fair bit to be desired in the bigger picture of Android’s Java-based ecosystem. What we needed was a user-compiled Swift shared library running on a non-rooted device from within a standard .apk application bundle.
The problem of Swift’s own dependencies
Up until that point, all the testing we had done was in the terminal of either a rooted device or an Android emulator. The reason being that Swift binaries would only run if the LD_LIBRARY_PATH environment variable was set to help locate Swift’s dependencies — its standard library, libicu and so on.
As we soon found out, the Android linker ignores LD_LIBRARY_PATH when loading libraries from within Java via System.loadLibrary(). Instead, it looks in its own system directories, and within the .apk bundle itself. The problem was, this didn’t work. Android’s error message was entirely unhelpful here too (unfortunately a recurring trend in our growing experience), stating that our Pitch Detection library was simply ‘not found’. Moving up the dependency chain and specifically loading various libicu libraries resulted in a message saying that these were ‘not found’ either. The catch was that certain libraries did load using this exact method, so we knew something was up.
After some trial and error and using some command line tools I’d never heard of before, we discovered that the load order of Android’s dependencies was important: it looked in its own directories first, before looking within the application bundle. It seemed Android was loading its own outdated version of libicu (which was missing the code Swift depends upon) and ignoring the version we were bundling in the .apk (I’m sure this was well intentioned, but seriously: why, Android, why??)
We tried to work around this for weeks, trying different solutions (parallel to testing Swift itself) and asking fruitless StackOverflow questions with no progress at all. In the end, the day after the Christmas weekend, we stumbled across a strange workaround in an obscure Google Groups post from 2011: rename the dependency files and use a text find-and-replace within the binaries themselves to update the dependency they were looking for. This would remove the version conflict with Android’s own version of libicu and allow us to load our own. The only catch was that, being binaries, we had to keep the strings we changed the same length. So libicu ended up being called libscu (s for Swift). So we ran
rpl -R -e libicu libscu libswift*.so
and ended up with what we needed– finally we could load Swift libraries on Android!
Testing our Pitch Detection library
By this point I was very happy to be all but out of the zone of compilers and binary tools and more into my comfort zone of working with Swift! And thanks to the prep work we’d done cleaning up our codebase for cross-platform use, it didn’t take long at all to get the first results.
Within 24 hours (albeit at about 4am from memory, note the f.lux warm display tint) we had an .apk running on a normal device, including a basic solution to the next challenge: making calls from Java into Swift and vice-versa.
My colleague Erik had already played around with some C(++) audio code on Android and had some callbacks to Java working via the JNI, so in theory it was just a case of calling the C JNI library from Swift code. To achieve this I initially just hacked the JNI headers into Glibc’s module.map as an add-on to the Android Swift Standard Library (I eventually cleaned this up and made a separate JNI module; see the next part of this series). This allowed us to call a Java method with no arguments (and no data), as seen above.
We needed a bit more control though: to truly interface with our web app (not forgetting the goal of releasing flowkey for Android!) we still had to use the JNI to pass around some strings and arrays to and from Java. The fact that I’d never used JNI before (or Java in any significant manner, for that matter) didn’t help. Neither did coding Swift in vim on a remote linux virtual machine — Swift for Android still today only compiles from a Linux host (nowadays we’re using vagrant). With no code completion, no debugger and Android’s inaccurate error messages, the next stage proved more challenging than I’d expected.
See here for the third part in the series: a technical introduction to calling Swift from Java and vice versa using the JNI.
