Native Android Libraries Gone Bad
Fixing UnsatisfiedLinkErrors when your native libraries unexpectedly start targeting new Android architectures
The Setup
I work on the Android app for a large company that is broken into multiple teams — Android, iOS, cloud services, etc. About a year ago one of those teams, the AV platform team, came out with an updated AV player library that we wanted to integrate into the app. Version 1.0 of the library was already present in the app so I expected the whole process to take about an hour. I just needed to copy over the updated AAR to the libs folder, fix any broken API calls, and call it good.
The Problem
So that’s what I did. I copied over the new library, updated a few calls, compiled, and ran the app. Everything looked almost right. The videos would play, but there were many spots in the app that just weren’t loading. Looking through the logs I saw:
java.lang.UnsatisfiedLinkError: dalvik.system.PathClassLoader[DexPathList[[zip file "/data/app/com.example.myapp-_aW5-UKF-hLJusoNIdxmNA==/base.apk", zip file "/data/app/com.example.myapp-_aW5-UKF-hLJusoNIdxmNA==/split_lib_dependencies_apk.apk", zip file "/data/app/com.example.myapp-_aW5-UKF-hLJusoNIdxmNA==/split_lib_slice_0_apk.apk", zip file "/data/app/com.example.myapp-_aW5-UKF-hLJusoNIdxmNA==/split_lib_slice_1_apk.apk"],nativeLibraryDirectories=[/data/app/com.example.myapp-_aW5-UKF-hLJusoNIdxmNA==/lib/arm, /data/app/com.example.myapp-_aW5-UKF-hLJusoNIdxmNA==/base.apk!/lib/armeabi-v7a, /data/app/com.example.myapp-_aW5-UKF-hLJusoNIdxmNA==/split_lib_dependencies_apk.apk!/lib/armeabi-v7a, /data/app/com.example.myapp-_aW5-UKF-hLJusoNIdxmNA==/split_lib_slice_0_apk.apk!/lib/armeabi-v7a, /data/app/com.example.myapp-_aW5-UKF-hLJusoNIdxmNA==/split_lib_slice_1_apk.apk!/lib/armeabi-v7a, /system/lib, /vendor/lib]]] couldn't find "my_lib_4.so"
at java.lang.Runtime.loadLibrary0(Runtime.java:1011)
at java.lang.System.loadLibrary(System.java:1657)
at com.example.myapp.MyApp.<clinit>(MyApp.java:35)
...The application suddenly couldn’t find some native dependencies that it had no problem finding before.
How can something like this happen? A little background on Android hardware is necessary to explain it. From Android’s guide on ABIs:
Different Android devices use different CPUs, which in turn support different instruction sets. Each combination of CPU and instruction set has its own Application Binary Interface (ABI).
A device’s OS can support one or more ABIs, and expects to find native libraries within the APK at the path /lib/<abi>/ where <abi> can be any of the ABIs listed here. The Android system will pick the ABI best suited to the device; it will first try its primary ABI (e.g. arm64-v8a), and if it doesn’t find anything it will continue to any of the secondary ABIs it supports (e.g. armeabi-v7a). The key thing to note is that the OS will only pick a single folder and will not combine the contents of two or more of the folders. Also keep in mind that certain ABIs are backwards compatible with others (armeabi-v7a and armeabi, for example). So native armeabi code can be run on armeabi-v7a devices, which will prove to be useful later.
I wanted to see how the binary was packaged before and after the library update. I gave the generated APK a .zip extension, unzipped it, and looked in the lib folder to see how the native dependencies were setup.
Before the update:
After the update:
The old version only contains native dependencies for the armeabi ABI. The new version now contains an additional target for armeabi-v7a. The best suited ABI for my device among the selections available is armeabi-v7a, so the OS is only looking inside that folder.
The AV platform libraries were compiled with the Android Native Development Kit, or NDK, which “…is a set of tools that allows you to use C and C++ code with Android.” Support for the armeabi ABI was removed in version r17 of the NDK. The AV platform team started using a version of NDK that no longer supported the architecture we needed. We could have asked the team to rollback their NDK version and supply us with an AAR that only supported armeabi, but other groups relied on this library and I didn’t want to be the one to block anyone else’s progress.
Possible Solutions
The best solution is for each native library in the app to target all of the possible ABIs we want to support. Unfortunately our application uses a large number of libraries from a wide variety of teams. Each library only targeted the armeabi ABI at this point. It would take quite a commitment from the entire global organization to go back and recompile everything we need. Some of those original teams probably don’t even exist! In the short-term this isn’t feasible, but as I discuss later, in the long-term this is unavoidable.
As mentioned above, we could also ask the AV platform team to rollback the version of NDK they were using. But, again, that was something I really wanted to avoid.
A last resort is to copy over all of those .so files into the application’s jniLibs folder. This folder sits alongside your Java or Kotlin code, and is a place where you can manually add native libraries, grouped by ABI.
If we copy over the contents of the libs/armeabi folder contained in the final APK directly into the jniLibs/armeabi-v7a here, everything would start working. The application would find everything it needed in a combination of the jniLibs/armeabi-v7a folder and the libs/armeabi-v7a folder the build process produces. However, any addition or future update to a native library is prone to break this. Other developers will need to know to unzip their new library, extract all the .so files from the libs/armeabi folder, and copy them over to the jniLibs/armeabi-v7a folder. It’s a lot of extra work for something that we can work around.
Our Solution
Luck is on our side in this particular situation. The new native dependencies we’re dealing with are targeting an ABI that is backwards-compatible with the other, armeabi-v7a and armeabi, respectively. As mentioned above, any device supporting armeabi-v7a is going to be able to handle native code targeting armeabi as well. As long as we can group all of these files together in one place for the app to find, everything should start working. So now the tricky part, how do we package these together?
Our overall goal is to take the files in one of the library folders and copy or move them over to the other folder. Which folder is the source and which is the destination doesn’t matter since we’ll be deleting the source directory, but let’s plan on moving everything over to the armeabi-v7a folder for demonstration purposes. The Gradle build process for Android is a powerful, if finicky tool, that we can leverage to accomplish this goal. The build process is broken down into multiple steps; for each step we can hook into immediately before or after it runs to run some code of our own. Each of these steps, or tasks, will have a name we can examine. One of these tasks, with a name prefixed by transformNative_libsWithMergeJniLibsFor…, is where these native dependencies first show up. This is where we start.
But first, a warning: even after many years of Android development, Gradle is still a mystery to me. I know enough to be dangerous, so beware, code that can definitely be improved ahead!
First we iterate through all of the build tasks. Anything with a name not starting with transformNative_libsWithMergeJniLibsFor we don’t care about, and can quickly bail on.
Now that we have the correct build task, we can get the system path where all of the build files are being placed. We iterate through all possible build variants of our application to compare against the version of the app currently being built. We do this by grabbing the portion of the task name after transformNative_libsWithMergeJniLibsFor, which will potentially contain the flavor and build type. If the flavor and build type of the variant being built are contained in this string, we most likely have a match. The variant’s directory name is a starting place to find the native dependencies.
In my case the task path is myAppFlavor1/release.
Finally, if the task path is found, we call the function to copy the files.
The copyNativeLibs function isn’t too complicated. First a “base” directory to where to find the native libraries is constructed based on the application’s project directory (a global variable called projectDir) and the task path that was just found.
Root directory to begin our search: ~/development/myApp/app/build/intermediates/transforms/mergeJniLibs/myAppFlavor1/release
From there we can search for the desired source and destination folders. Note that once we find the source directory (armeabi), we immediately construct the destination directory (armeabi-v7a) whether it already exists or not. No matter what, we copy files into that directory.
Source directory to copy from: ~/development/myApp/app/build/intermediates/transforms/mergeJniLibs/myAppFlavor1/release/folders/2000/1f/main/lib/armeabi
Destination directory to copy to: ~/development/myApp/app/build/intermediates/transforms/mergeJniLibs/myAppFlavor1/release/folders/2000/1f/main/lib/armeabi-v7a
Once we have both directories, we copy over all native dependencies (files with the extension .so), and then delete the source directory. We can just leave the source directory alone, but those files will never be used and will just contribute to an increased APK size. I use the Apache Commons IO library to copy and delete the directories, but you can probably use the Gradle Copy/Delete type tasks or come up with something of your own if you didn’t want to import a third-party library.
Here’s the function to search for a sub-directory referenced in the previous code snippet above. It just iterates over a directory’s list of files, looping recursively over any sub-directories it might find, until it finds the desired directory name.
The new folder will get rolled into the APK just as if the build process itself had constructed it. Once the build finishes and we have the final APK, we can unzip it to verify the directory structure is to our liking:
Notice that there’s only the single armeabi-v7a folder now, just as we want. A quick test confirms that the app is now running as expected.
What has happened since
As mentioned above, the solution I went with is not the long-term solution required. We need to update all our native libraries to support more than just the armeabi architecture (and even beyond armeabi-v7a). Besides eliminating the need for the work-around I implemented here, it has two main benefits:
- By supporting the x86 architecture, we will be able to deploy the app to an emulator. You can setup an emulator to target a different ABI than x86, but I wouldn’t recommend it. Everything from system startup to running an app is preposterously slower. Also, without emulator support it’s next to impossible to run UI tests in a continuous integration environment. From my point-of-view as a developer, this would be the biggest benefit.
- It is now required! In an article from the Android Developers Blog, starting August 1, 2019:
All new apps and app updates that include native code are required to provide 64-bit versions in addition to 32-bit versions when publishing to Google Play.
So at the very least our libraries need to support the arm64-v8a architecture in addition to armeabi; otherwise we won’t be able to update our app in the Play Store. Fortunately, in the year since I had to implement this solution, the various teams have been able to add 64-bit support to their respective libraries. We don’t have x86 support yet, but we’re at least able to update our Play Store listing. That’s at least a start, and will hopefully lead to support for a wider range of ABIs in the not-too-distant future.
Sean works at Livefront, where he makes working with Gradle harder than it should be.
