How Google (and Amazon) could make Android developers happy

I hit a few surprising roadblocks writing and publishing an Android game, some easily fixed.

My friends at inkle studios are half-way through a beautiful iOS adaptation of Steve Jackson’s Sorcery!, a 4-part gamebook series. I recently ported part 1 to Android, which was a really fun and challenging project—see inkle’s blog post for the details. Based on that experience, I’d like to suggest a few things about Android development that could use improving.

This is going to get pretty technical, but I’ll try to explain all the acronyms I use. Anyway, here’s my wish list:

1. Make incremental development faster
2. Make a better build system for the NDK
3. Make it easier to upload builds to the store
4. Let me give the app to testers (and reviewers) for free
5. Fix expansion files
6. Fix OBBs
7. Provide tools to generate JNI bindings
8. Fix audio
9. Make it easier to set the app’s price
10. Make it deploy instantly

I know, I could have called this article “10 things Google should fix about Android”, but I don’t work for Buzzfeed.

Note that our app is a game with lots of graphics and sound, and uses the NDK (the Native Development Kit, meaning it’s written in native code rather than Java). We’re also selling the game for money rather than relying on ads or in-app purchases. Some of these points won’t apply to other types of app.

I’ve been involved in Android development before but this was my first time making a complete app. If I’m going to make more Android apps—and there are 3 more Sorcery! episodes coming, so that’s the plan—I’d love to see some of this stuff fixed. This is my attempt to kick up a fuss and maybe make something happen. And maybe to get a rant or two off my chest. Let’s get started!

Make incremental development faster

This one probably isn’t easy to fix, but it’s the one I’d most like to see.

Before getting stuck into the Android port I had to get familiar with inkle’s iOS codebase, so I spent a lot of time inside Apple’s development environment, Xcode. There are definitely some problems with the iOS development flow—for example, sorting out the certificates so you actually have permission to run your own code on your own device—but once you get going it’s pretty good. Builds are fast, running your code on the emulator is fast, deploying to a real device is fast (even for big projects), and the debugger works well.

On Android, in contrast, builds are a little slow (see the next topic), and deploying to the device is very slow. Whereas Xcode seems to just sync modified files, on Android you need to zip everything up into an APK (Application Package) and push the whole thing over with “adb install” (adb means Android Debug Bridge; it’s a script you use for low-level access to your phone from your computer). If your APK is big this can take a minute or so. You can move some resources out of the APK, but that also comes with headaches—see “fix expansion files” below.

And adb is flaky! It’s a really thin protocol for connecting to your phone over USB, so I don’t understand why it’s so bad. It varies a lot across devices. My first-gen Nexus 7 loses its adb connection about half the time, and my old Galaxy Nexus takes about a minute to connect. I suspect people just think oh, well, USB is flaky sometimes and put up with it. But that’s not true! Otherwise your mouse and keyboard and external drive would fail constantly, which they clearly don’t. There’s something dodgy about adb and it needs sorting out.

If you want to debug your code while it’s running on the device, that’s an extra layer on top of adb, and for NDK projects at least, it’s even more flaky. It would be nice to be able to debug code running in an emulator, but the stock Android emulator is very unstable on my Mac, and far too slow to be usable.

As a result of all this, I did most of my coding and debugging in Xcode (on an iOS device; most of our codebase works on both platforms).

There’s a third-party emulator, Genymotion, which works great and is really fast. I haven’t yet checked whether it’s debuggable, but I could potentially see myself doing most of my Android development via Genymotion. However, you’ll always need to test your code on a real device at times, particularly in the final testing and bugfixing phase before launch, and that means using adb.

I briefly tried the new Android Studio, but it looks like it doesn’t fully support the NDK yet. The development workflow for Java apps might be a lot better than I’ve described, but I’d really like to see the NDK get some love. On which note—

Make a better build system for the NDK

To build an NDK project you use the “ndk-build” script, which wraps a bunch of Makefiles. Makefiles! And highly complex recursive ones, those are just the best. So the build is slow, and frequently gets confused so you need to “make clean” and start again. The NDK badly needs a good build system. Nobody uses CVS for source control any more, so why do we still use a build tool from the 1970s?

I’ve been using Facebook’s Buck build system, which I was really excited about because it’s a clone of Google’s proprietary build system, which is the best I’ve ever used—it’s pretty fast and always correct, which is really liberating once you learn to trust it. Anyway, Buck is okay but not quite there yet; and for an “ndk-library” target it does a little prep work (which is pretty useful) but then just runs “ndk-build” under the hood. Oh well. At least it means you don’t have to use Makefiles for the rest of your build.

The NDK does actually have an option to generate a “standalone toolchain”—an installation of cross-compilers and linkers and header files and so on that you can invoke individually, like you’d normally do in Unix. In principle you could hook that up to a proper build system (Ninja, maybe?) and ditch the Makefiles entirely. Somebody should do that.

Make it easier to upload builds to the store

The Play Store lets you upload new builds any time you want, with separate alpha/beta/production labels. This is great, much better than the Apple and Amazon approach of “send us your app when it’s finished, and we’ll put it on the store for you in a few days or weeks.”

However, the actual upload process is not so great, as you have to do it via a web page. Maybe fine for a 5MB Java app, but tedious for a 150MB game.

The fix for this one is easy: let me provide a URL for the build, rather than having to upload it via the web. I put all my builds on DropBox (much more reliable than a web app), and Google should be able to just slurp it over from there. There’s no security problem, as it’s exactly equivalent to me downloading the file and uploading it again; it just cuts out the middleman.

Same goes for Amazon. To upload an APK to their store, you either use a web app, or SFTP(!). I don’t know what the Apple store is like, but I wouldn’t be surprised if it’s the same sort of deal. (Actually, inkle’s Joseph Humfrey tells me the process is very easy! You just select “upload build” in Xcode—of course.)

Let me give the app to testers (and reviewers) for free

This one is just bizarre. When you first start a new project on the Play Store, you have to choose whether it’s paid or free. And this decision is permanent: once you make an app free, you can never go back. OK, fine, I dislike the entire free-to-play model, so ours is a paid app.

Then you get a build that mostly works and you want to show it to some testers. You can use Google Groups to add testers to your Play Store account, great! But, the testers have to buy the app. I can’t give them vouchers, I can’t make the app temporarily free. I could reimburse them, but Google takes a 30% cut! It’s just ridiculous, and surely very easy to fix.

The same problem arises when you’re close to release and you want to send a build out to reviewers. I’d love to leverage the Play Store for that—free analytics, so I can see how many reviewers actually installed it!—but unless I want to charge reviewers money, I can’t.

I don’t think Google have properly thought through the workflow for paid apps. It’s a shame, because we deserve alternatives to ad-supported apps or microtransaction hell. Those models have their advantages, but the thing about a paid app is that its only incentive is to give you a really good experience (in the hope that you’ll recommend it to your friends), rather than waste your time or upsell you.

There are some great third-party alternatives for sending builds to your testers, but I didn’t realise this until too late. I almost went with TestFlight, but then Apple bought them and they coincidentally stopped supporting Android. I’ve just started using HockeyApp and it looks very promising.

So what we actually did was send people emails with DropBox links. That worked fine except for another Play Store twist, expansion files.

Fix expansion files

The Play Store doesn’t allow APKs larger than 50MB. For a game like ours with lots of graphics and sound, that’s not very much—Sorcery! is around 180MB, and there are plenty of games on the Play Store that are even bigger. The extra data needs to be split into one or two extra “expansion files”, which can be much bigger (500MB each, I think).

This isn’t intrinsically bad, just weird. It’s a leaky abstraction. Shuffling massive files around in flash memory is awkward, sure, but why should developers be forced to deal with that, and via such a clunky mechanism? Why can’t the Android platform deal with it for us, and hide that complexity?

On iOS, I neither know nor care what’s really going on under the hood, but my app is a bunch of files, and when the user installs it they get the same bunch of files. (In fact I imagine it is just a bunch of files; iOS has always had a reasonably good filesystem, while Android originally used FAT and has been backing away from that decision ever since.)

On the Amazon store it just works too! They don’t have an APK size limit, so you can split it across several files, or just make one massive blob, whatever you prefer.

So the easy way to fix this is just to remove the APK size limit.

You could argue that that’s bad, because it forces users to download a massive APK from scratch on every update. Well, fix that too! Just check what the user currently has installed, and send them a patch to update it to the new version. Google already does exactly that for Chrome. (I don’t actually know whether the Play Store uses smart diffs, but when you download an update it reports the full app size in the progress bar, so it looks like it’s just downloading the entire thing from scratch.)

As noted above, I used DropBox to send out test builds of Sorcery!, so that meant I had to write some code to download and install the expansion file from DropBox too, attempting to be compatible with the way the Play Store does it. Google has an official Expansion File Downloader library, but it looked difficult to unpick it from the Play Store—no obvious way just to plug in my own URL. So I initially used Android’s built-in DownloadManager, which turned out to be a mistake because DownloadManager has some nasty bugs. Maybe that should be another topic, but this article is getting pretty long already…

For example: use DownloadManager to download a file into your private directory. During the download, uninstall the app. On my test devices at least, the download continues and you get a file where your app’s private dir would be. And if you reinstall the app you get a different user ID (I think this is a KitKat security feature), so that leftover file will cause you a lot of trouble.

These aren’t fundamentally tough problems, just tedious. They should be fixed transparently at the platform level, not foisted onto developers.

Why is there an Expansion File Downloader library at all? Supposedly the Play Store may in some circumstances install just your APK and not the expansion file, so your app must be prepared to install it manually. I think the idea is to avoid massive downloads over mobile data. But in KitKat, it always seems to download the expansion file first, so this situation never arises. Did somebody in the Android team realise what a headache the whole expansion file business is and just fix it up front? If so, great! Even if it means I wrote and debugged some code that will never actually be needed—it shouldn’t be needed.

Fix OBBs

So what is an expansion file? The filename has to end in .obb, for Opaque Binary Blob. It can just be an arbitrary blob of data, but in fact it’s usually a disk image (calling that an “OBB” seems to be a Linux quirk). You ask Android’s StorageManager to mount the virtual disk, and there are all your loose files, just like I wanted earlier!

Unfortunately this system is not reliable. It broke completely in the first release of KitKat, so I assume there wasn’t a regression test. There were serious bugs in JOBB, the tool you use to build an OBB in the first place. Then when I finally got my OBB mounted, it didn’t actually work very well! It’s dangerously easy to get into a state where your OBB is mounted, but you can’t read it or unmount it. (I suspect KitKat security improvements are behind this again.) The only solution I found is to reboot the phone, but if the user has to reboot to use my app, they’re going to blame my app, not the OS.

In the end I ditched the whole disk image idea, and just used an actual binary blob. I concatenate all my resources into one big file, along with an index to locate the right data chunk for each filename. (I could have used a zip file, I just happened to have written some index code already.) It works. Faster than mounting the disk image, too.

So I don’t care as much whether this one is fixed, but it would be nice to have that disk image as an option. Sometimes having real files and real filenames is useful for organising your data (especially if you’re relying on third-party code that makes this assumption). As I said before, an app is just a bunch of files, and the platform should help me get those files onto the target device as transparently as possible. This should be a relatively simple feature and there’s no reason for it to be unreliable. (The actual reason is probably FAT again.)

Also, if a feature is documented in such a way that it seems to be a normal and fully-supported part of the platform, it should work. I’m sick of APIs like DownloadManager and StorageManager that break when you put a little weight on them.

Provide tools to generate JNI bindings

Speaking of StorageManager, I initially tried to use it directly from C (i.e. native code) rather than Java, since there’s a C wrapper for it in the NDK. Unfortunately the C wrapper doesn’t work. For example, one function returns a string; and in C that means you need to be careful about managing the memory. Who owns the string? Turns out nobody owns it.

The root problem must be that somebody wrote a JNI (Java Native Interface) wrapper by hand, which is fiddly and error-prone at the best of times, and (again) there wasn’t a regression test. So I had to bypass the NDK’s C interface and call through to Java, which means I had to write my own fiddly, error-prone JNI wrappers!

The thing is, writing a JNI wrapper is strictly a mechanical task. Bugs only slip in when your fingers get tired and you make a stupid typo. If there were a good tool for generating wrappers, it could solve both problems—Google could use it to generate all the NDK wrappers, and app developers could use it to wrap their own classes. And again, this is something the Chrome team already does! There are some Python scripts that just need to be unpicked from the Chrome codebase and generalised.

There are tools like SWIG, but they all seem to be focused on wrapping C/C++ so it can be called from other languages. Working with NativeActivity and the NDK, I mostly need to go in the other direction, calling Java from C, and to my surprise I couldn’t find any good wrapper generators.

Apportable have something they call BridgeKit for calling Java classes from Objective-C, and it’s really terrific (though I don’t know if the wrappers are generated or hand-written). I evaluated Apportable but didn’t end up using it, as Sorcery! relies on some UIKit features they didn’t yet support; BridgeKit is one of the main Apportable features I miss.

Fix audio

This is an area where Android is definitely behind iOS, which has excellent and robust support for low-latency audio, with both high-level and low-level APIs. Android traditionally just has high-level, high-latency Java APIs. (That’s why the good music apps are mostly on iOS.)

Android also has something called OpenSL ES, a low-level audio API by the same group who maintain OpenGL. This has been available for a while (since Gingerbread), and the idea is that it can be both hardware-accelerated and low-latency. It’s kind of awkward to use, but there are a bunch of higher-level audio libraries that can sit on top of it, so that’s not a problem.

Unfortunately it seems like OpenSL still isn’t ready for prime time. It just doesn’t work consistently and reliably across all devices.

For Sorcery! I use Cricket Audio, a nice simple cross-platform library. By default Cricket uses OpenSL, so I got weird buzzy audio on certain devices, like the first-gen Nexus 7. But luckily, it just needed a simple flag change to switch to Java audio instead, and that seems to work fine. I’m so glad I used a library rather than relying on the OS!

When I mentioned this problem to Cricket, they quickly pointed me to an Android bug report from last year, with a patch just waiting to be submitted. They’re also going to work around the problem in the next Cricket Audio release.

So, what’s to be done? If OpenSL can be made reliable, that would be great. But a lot more development effort goes into OpenGL ES, and that’s still distinctly shaky in some areas (on all platforms, not just Android) so I don’t know how hard it would be to “just fix” OpenSL.

Make it easier to set the app’s price

Back to the store, and another easy one. When setting a price for your app, the only options seem to be:

• Set a single price and base currency, and have the prices for other countries calculated automatically;
• or set all the prices manually, for 100-odd countries.

Oh, and there are checkboxes if you want to set some prices manually, but that seems even more fiddly and error-prone than JNI.

Amazon does the right thing here: you manually set a price for one or more currencies (not countries) that you care about, and anything you don’t specify is set automatically. So we were able to set a nice round price in pounds, dollars and euros, and be confident that prices in other markets would at least be comparable.

If you really want fine-grained control over the price in specific regions, I think Amazon lets you do that, but it doesn’t force you to do that.

Make it deploy instantly

Last one. As I mentioned right at the start, the basic Google Play Store workflow is much more developer-friendly than either Apple’s or Amazon’s stores. You upload builds whenever you want, and you can use the store to distribute builds to your testers before release.

Except… When you press the “submit” button, it says “changes may take a few hours to appear in the Play Store.” Why? It would be really great if my changes were made available instantly. The current behaviour isn’t terrible, but it means if I upload a new build I don’t know exactly when my testers will get it. (As noted above, in the end we mostly just used DropBox for testing.)

Maybe there’s some deep technical reason why instant uploads would be too expensive, due to content caching or some such—I don’t know, I’ve never built a system like that. But it reminds me of an interesting theory I’ve heard for why YouTube won the video wars (over the likes of Google Video and Metacafe): when you uploaded a video, it appeared instantly, whereas all other sites took a few minutes to process the video. Users loved that, and supposedly that’s one of the reasons YouTube took off. If that logic worked in 2006, and was technically feasible in 2006, why not in 2014?

Note that I don’t mean new builds should actually install to user’s phones instantly; just that if I say “hey, there’s a new release!” and they open the Play Store app, they should see the new build.

Conclusions

For platform developers, the executive summary is my original list of 10 points. Fix those, please!

For app developers, well, the lesson I draw is that Android has lots of great features but it’s also rather fragile, in ways the docs wouldn’t lead you to expect. It’s a good idea to use a third-party toolkit if you can, rather than building directly on the Android platform. (Again, things may not be so bad for pure Java apps, rather than NDK apps.) For your application framework, there’s Apportable, and I also hear good things about Cocos2D, Unity and Corona. For incremental development, Genymotion is far and away the best emulator. For crash reporting, use ACRA and one of its many supported backends (I’m really liking HockeyApp).

I’m happy with my decision to use a low-level approach for Sorcery!—the GNUstep Objective-C runtime and Foundation framework, with hand-written versions of the other frameworks—but only because I started with a large and sophisticated iOS codebase, with three additional episodes still to come! The port took a long time, but rewriting ~5 person-years of work in another language would also have taken a long time, without any notable savings for future episodes.

Also, it was fun to write! Despite tearing my hair out at times over some of the platform bugs mentioned above, I learned a lot about Android and OpenGL in the process.