Migrating A large Android Application From Ant to Gradle
This post describes our experience migrating the KeepSafe Android app to the new Gradle-based build system. This is not meant to be a step-by-step tutorial.
In this post, I want to focus on some specific cases we have had problems with. This also represents the state of the build tools at the time of writing. Some of it might be outdated and no longer valid.
Why move from an existing solution?
When starting to build an application, it’s easy to pick the tool that least gets in your way. With KeepSafe, we started with the default choice at the time, which was building the project with Eclipse/Ant. This choice ended up not being ideal, however. As we grew and more and more people started working on the same thing, managing many different Android library projects became a massive pain. We realized we needed an extendable, IDE-agnostic build system with dependency management that was also easy to use with CI.
Our options
There are several tools on the market that allow you to build an Android app. We spent some time investigating how each of them would fit in our environment:
- Makefile/Scripts — the most flexible but also the most time consuming solution. The lack of dependency management makes this option no better than what we already had.
- Ant — What we already had. Dependency management can be added through Ivy. It’s easy to write custom tasks, but it’s ugly and verbose. The more you need, the faster the scripts grow, and after a while you end up with thousands of lines of XML.
- Maven — Seems to be the most used solution for building Android apps. While Maven is supported by most tools, doing anything custom is a pain. Also, it’s still XML.
- Gradle — Picked by Google last year as the default Android build tool. It’s progressing quite fast, but it’s still missing features, and the API is not stable. Gradle has a nice DSL and it’s easy to extend and integrate with other tools.
- Buck — Created by Facebook. It’s hard to judge this tool without using it. Seems to a be niche tool with very specific use cases according to the documentation: > Buck is designed for building multiple deliverables from a single repository rather than across multiple repositories.
Our decision
For us, it really came down to Maven and Gradle. While Maven is more mature, Gradle seems to be gaining ground quickly, and because it’s backed by Google, we can expect good, long-term support. It’s also the default build tool for Android Studio, it has some great features, and it isn’t XML-based.
The migration process
apk, apklib, aar — a whole bunch of different library formats
In the Android world there have been many different file formats to include libraries and Android library projects. So far there have been apk files for the actual app, jars for normal Java libs and apklibs for android library projects. Unfortunately, those are not compatible with each other.
Google decided to introduce a new format for Android libraries, .aar. This is way nicer than having to link to the actual Android library project in Eclipse and makes versioning very easy.
The old Maven plugin format,apklib, is not supported by Gradle. This means we have to migrate the existing projects that offer apklib as a format into aar lib files. Our solution was to build aar lib files for those projects ourselves.
Main app
If you’re using the default Android Developer Tools (ADT) in Eclipse you’ll find an option to export you current project to Gradle (only in ADT version 22.0 or higher). ADT itself does not support Gradle right now, so you will be better off switching to Android Studio. You can try to use Gradle outside of ADT on the command line or use the default Gradle plugin for Eclipse, but I didn’t have much pleasure/luck with that. You have to do some wonky symlink stuff for Eclipse to find your resources and other things.
There are several differences between the ADT Eclipse and the Gradle build system. The biggest two differences are the project structure and the test project integration into the main project. If you don’t want to move to the new default gradle structure you need to tell Gradle where the files are located. In our case:
android {
...
sourceSets {
main {
java.srcDirs = ['src']
resources.srcDirs = ['src']
aidl.srcDirs = ['src']
renderscript.srcDirs = ['src']
res.srcDirs = ['res']
assets.srcDirs = ['assets']
jniLibs.srcDirs = ['libs']
manifest.srcFile 'AndroidManifest.xml'
}
debug {
java.srcDirs = ['debugSrc']
res.srcDirs = ['debugRes']
}
release {
java.srcDirs = ['releaseSrc']
res.srcDirs = ['releaseRes']
}
androidTest.setRoot('test')
androidTest {
java.srcDirs = ['test/src']
res.srcDirs = ['test/res']
resources.srcDirs = ['test/src']
manifest.srcFile file('test/AndroidManifest.xml')
}
}
...
}This is roughly what we ended up with. The test project AndroidManifest.xml is usually auto-generated, but if you have something specific, you might want to use your own.
Spec everything
We’ve learned that you want to spec as much as possible in your build.gradle file. This is one of the main advantages to get consistent builds across different environments. We define our Java compiler, and the build tools and SDK versions.
tasks.withType(JavaCompile) {
options.encoding = "UTF-8"
sourceCompatibility = "1.6"
targetCompatibility = "1.6"
}
android {
compileSdkVersion 17
buildToolsVersion '19.0.1'
}Libraries
As I’ve already mentioned, there is a new Android library format .aar. If you have your code split into more than one project like us, you will need to migrate that as well. It’s not much different from migrating the main app. There are, however, some problems you should be aware of.
Build types
There are 2 default build types: debug and release. Unfortunately, this setting does not propagate when building your main app in debug. Even if you have your dependencies set up correctly and you try to build debug apk, it will include release aar. This means you want to have all of your configuration in the main app. We already had an existing library with debug settings. We ended up moving all of the library configurations that are dependent on the build settings to the main app to configure the lib at runtime. One example is debug logs. We are now initializing our internal lib with the log configuration from the main app instead of with the config from the build type of the lib. There is an issue #1 and #2 opened for this already.
Maven and aar
There is some work done to be able to create aar from Maven:
- https://github.com/jayway/maven-android-plugin/pull/228
- https://github.com/jayway/maven-android-plugin/pull/227
It was not stable enough when we were trying to use it, so we ended up creating Gradle build scripts for the libraries we were building and including those in our CI.
Build types & signing
We have 3 different types: debug, beta and release. You can have different sources, resources and so on for each build. This is quite useful if you want to configure some services, endpoints, or different icons for debug build. These are working quite well and we had no problems here. If you want another build type, the easiest way is to inherit it from an existing one.
beta.initWith(buildTypes.debug)
beta {
runProguard true
proguardFile 'proguard.cfg'
versionNameSuffix "-beta"
packageNameSuffix ".beta"
}The build types we ended up with are:
Debug — has to run as fast as possible as we run it all the time during development. This means we don’t sign the APK or run code optimization tools like Proguard.
Beta — a similar build to our release build, just without signing the APK at the end as we don’t feel comfortable having our Google Play store key password flying around. We load the same config as for release and also run Proguard during the build. This is the build type we run on our CI server, as it will detect errors around Proguard configurations that might have slipped during development.
Release — This is the final version that we shipped to the app store. Because the release apk also needs to get signed, we wanted to make sure that the password for our release keystone are never stored anywhere besides our password managers. So we pass it in as a parameter when building with the following settings under signingConfigs:
release {
storeFile file("release.keystore")
storePassword project.hasProperty('storePass') ?
project.storePass : "default_pass"
keyAlias "release"
keyPassword project.hasProperty('storePass') ?
project.storePass : "default_pass" }It takes the password from the command line with -P<property_name>=”<property_value>”. You can read more here
gradle assmbleRelease -PstorePass='password'If the password is not set for assembleRelease we throw an error:
gradle.taskGraph.whenReady { taskGraph ->
if (taskGraph.hasTask(':assembleRelease')
&& !project.hasProperty('storePass'))
{
throw new IllegalArgumentException('Run with "-PstorePass=
<value>" to sign the release build')
}
}There are a few other options on the web for getting the password from the console. The problem we had was an error if we tried to change it after the build had started, like getting it from System.console(). Gradle complained that the password had been messed with. This seems to be new with the 0.9.* release of the build tools.
Smoke/sanity tests & testing Proguard
Ideally we would like to run smoke/sanity tests on our release apk, but since we need to sign it separately, there is no way to do it with CI. For this reason we use our beta build that has Proguard enabled. You can choose which build type is used with the integration tests by specifying it in your script:
android {
testBuildType "foo"
}More details on this can be found here. We’ve ended up creating a separate Gradle file, importing the standard build, and overriding some values:
apply from: 'build.gradle'
android {
sourceSets.androidTest.setRoot('smokeTests')
sourceSets.androidTest {
java.srcDirs = ['smokeTests/src']
res.srcDirs = ['smokeTests/res']
resources.srcDirs = ['smokeTests/src']
manifest.srcFile file('test/AndroidManifest.xml')
}
}NDK
The Gradle plugin is able to run NDK. This was not officially supported when we were migrating to Gradle. For this reason, we ended up checking the compiled binary files (*.so) into our repo and including them with the build:
android {
sourceSets {
main {
jniLibs.srcDirs = ['libs']
}
}
}Full NDK support is coming and should work out of the box as soon as it’s stable in the plugin.
Package suffixes
Being able to have different package suffixes is something I’m very excited about. This finally allows us to have a debug build in parallel to the production version on one device. You can have a package suffix for different build types like this:
android {
buildTypes {
packageNameSuffix ".beta"
}
}Remember to not hard-code the package in your xml files. Instead use:
xmlns:header="http://schemas.android.com/apk/res-auto"Additional info
Gradle tasks
The Android Gradle plugin updates the task graph dynamically. This means you can’t reference tasks in your scripts directly like:
task.doLast()Instead you need to do it after the graph has been created:
gradle.taskGraph.whenReady { taskGraph ->
...
}Some tasks might be included depending on your configuration. For instance, the preDex task won’t be included if you configure your build to run with proguard. It’s what it is :(
Performance
One of the major disadvantages of switching to a Gradle build system is increased compile/build time. However, we were able to speed up the compile time for a gradle clean assembleDebug build by a significant amount by adding
DEFAULT_JVM_OPTS="-Xmx512m"to our gradlew wrapper file.
We also use Facebook’s fork of proguard that is, according to Facebook, about 2.5X faster than the original Proguard implementation while maintaining identical binary output.
Several different integration tests
We wanted to have separate smoke/sanity tests; tests that generate screenshots, and normal tests run separately. The simplest way we were able to find to do this was to create a separate build file and override the test configuration:
android {
sourceSets.androidTest.setRoot('smokeTests')
sourceSets.androidTest {
java.srcDirs = ['smokeTests/src']
res.srcDirs = ['smokeTests/res']
resources.srcDirs = ['smokeTests/src']
manifest.srcFile file('test/AndroidManifest.xml')
}
}This is far from ideal, but it gets the job done.
Trying to be dynamic
A few things we tried to make work dynamically didn’t pan out; e.g. getting the version from the manifest…
The mix of setting the data during the script compile time and generating the task graph dynamically somewhat limits flexibility. We ended up hard coding some of those things. This might not be as elegant or as concise as we would like, but it works.
Linting
If you haven’t used lint on a regular basis, it can be a pain to remove all errors at once. To skip failing on error add:
android {
lintOptions {
abortOnError false
}
}Unused resources
We’ve created a separate tool to remove unused resources reported by lint. Check out our android-resoruce-remover
Bugs
Some bugs you might be interested in following:
Originally published at keepsafe.github.io on June 4, 2014.
