APDE GSoC ’18: Android Mode 4.0 Integrations
Hello! My name is William Smith and I am a student for the Processing Foundation taking part in the 2018 Google Summer of Code.
My project is the continued development of APDE, an Android app for creating Processing sketches directly on an Android device. I am the primary developer of APDE, but this year, the Processing Foundation’s sponsorship of my project through GSoC is allowing me to get a lot of things done that I would not have otherwise had the time for.
Briefly, the planned features for this summer are:
- Android mode 4.0 integration (wallpaper, watchface, and virtual reality (VR) support)
- Sketch preview mode (run sketches without installing them)
- In-app tutorial system (for users new to APDE)
- Incremental compilation (show errors and warnings in real time)
As you may have guessed from the title, I am nearing completion of the Android mode 4.0 integration. I am also on schedule! (fingers crossed)
Quick Links:
- Google Play (download)
- GitHub (open source, wikis, issues, download)
- Preview Channel (preview releases)
What’s New?
APDE can now run sketches targeting regular Android apps, live wallpapers, Wear OS watch faces, and Google VR (Cardboard, Daydream, and magic window). Previously, APDE could only make regular Android apps.
For the user, these features are available from a new button and dropdown menu next to the run button:
Wallpapers work like this:
When you run a sketch with the watch face target, the sketch builds and gets sent to the watch. Then the APDE wear companion app automatically loads the sketch and launches the watch face chooser:
VR sketches are installed just like regular apps. They support Google’s VR platform, which includes Cardboard and Daydream headsets:
If you want to get your hands on these new features, please join the APDE Preview Channel and follow the instructions at the top to join the tester list.
As of writing, v0.5.0-pre2 is available and includes wallpapers and VR, but not watch faces.
How does this all work?
The key to supporting these new targets was upgrading to use desktop Processing’s Android Mode 4.0, which added support for wallpapers, watch faces, and VR. My work was just porting these cool new features over to APDE, which was not itself an easy task.
I decided early on that I want every sketch to be runnable for every target. I want users to be able to take any sketch or example and seamlessly transplant it from one environment to any other, be it app, wallpaper, watch face, or VR. This way there is maximum flexibility and capacity for exploring the full potential of Processing sketches. The only exception to this rule is that VR sketches must be 3D (2D sketches just don’t work due to the nature of VR).
This approach differs from that of Android mode. For example, on the desktop, VR is packaged as a library, so only sketches that explicitly import the VR library can be run for the VR target. In APDE, every 3D sketch can be run for VR and the correct imports are added automatically.
The build system now uses a set of templates to keep track of all of the build files like the activities, manifests, and layouts. This method is much preferable to the hard-coding that was done before. The templates are taken pretty much verbatim from Android mode, but there are a couple of changes, such as support for Daydream VR headsets.
Live wallpapers and VR sketches are installed just like regular apps sketches, but VR requires the Google VR libraries. Similarly, watch faces require the wearable support library, but they do not install the same way because they must be installed on a watch.
Watch Face Woes
I needed to install the sketch onto the watch.
The first thing I tried was pushing the sketch to the watch and installing it there. It turns out that side-loading an APK onto a Wear OS (formerly Android Wear) watch is not as easy as it may seem. On an Android phone, you just need to enable installation from unknown sources in settings and you’re all set. On wear, however, this installation is blocked by the system and there is no way around it without rooting either the phone or the watch or connecting the watch to a computer. None of these are options for an entry-level mobile development environment.
Next, I tried packaging the app as a Wear 1.x app and installing it on the phone. Before Android Wear 2.0, all Wear apps were distributed by being bundled in phone apps, even if that phone app was just an empty shell. This all changed in 2.0 when wear apps became standalone. The hope was that I could take advantage of the old system to install the watch app through the phone, but this was to no avail. I still do not know whether or not the old installation method is supported on newer watches, but I was unable to get a proof of concept to work.
Finally I resorted to a last-ditch method: class loading, which is a kind of black magic. The idea is that instead of installing the sketch as an independent app, I can load the sketch’s class into an app that is already running on the watch, removing the need to install the sketch each time that it is run. This is a fundamentally different model from how APDE currently works. APDE builds a sketch and it is a standalone app that gets installed. The apps screen accumulates a bunch of sketches as the user runs them and they stick around when APDE is uninstalled. The watch is different. Now sketches aren’t installed, they are just loaded. There is only ever one sketch available as a watch face, and it disappears when APDE is uninstalled.
It turns out that this approach works. The user installs a bridge app on their watch from the Play Store (not available yet) called the “APDE Wear Companion”. When APDE builds a sketch, it sends it to the wear companion. The wear companion unpacks the sketch and prompts the user to select a watch face built into the wear companion. Then that watch face loads the sketch and displays it. This is surprisingly seamless.
There were a couple of challenges.
When the sketch crashes, it is running in the wear companion, so the whole wear companion app crashes, too. I am currently trying to contain the damage, but sometimes the wear companion breaks and needs to be restarted because the sketch crashed too many times. APDE also can’t see the stack traces from crashing watch faces at the moment because the process dies before the stack trace can get sent out. (Print statements work though.)
There isn’t just one watch face; there are actually two. Processing has two base classes for watch faces: one for the default renderer (JAVA2D) and one for OpenGL-based renderers (P2D and P3D). Thus there are two watch faces, one for each renderer. When the wear companion unpacks a sketch, it checks to see which renderer it is using. Then the corresponding watch face is made visible and the other one is hidden. This is another kind of black magic because the user should not even notice, but it seems to work fairly well.
I just lied. There are actually four watch faces. It turns out that when you load a new watch face, if the new one is the same as the old one (and all sketches use the same two watch faces), then the system “short circuits” (that is the technical term for it) and doesn’t reload the watch face. This is fine and dandy for regular watch faces, but the APDE watch face needs to reload in order to get the new sketch. The solution is dirty, but effective: just have two watch faces for each renderer (four total) and alternate between them when loading new sketches so that the watch face is forced to reload.
Despite all of these difficulties, I am happy to say that watch faces seem to be working fairly well, at least on the two sets of devices that I have tested them on. The other, more-likely-to-be-used targets — wallpapers and VR — were significantly easier to implement.
Looking Forward
The next major feature planned for this summer is a “preview” mode. Currently, APDE sketches need to be installed every time that they are run. This arrangement is non-ideal because the installer doubles the time between pressing “run” and seeing the sketch on some phones. Also, as described above, all of the sketches can clutter up the apps screen. Not having to install sketches is probably APDE’s most-requested feature.
I originally had two proposed solutions to this problem. First, use Processing.js to convert the sketch to JavaScript and run it in a WebView. Second, take advantage of Instant Apps to run a sketch without installing it. Both of these solutions are still imperfect. Processing.js is out of date and doesn’t support any native Android APIs, such as the accelerometer. Instant Apps is a shaky idea at best because I don’t even know if it will be possible to spoof the debugger to host the sketches locally and run them, or if this would be faster than just installing them.
Thankfully, cracking the watch face nut has given me a new approach. APDE doesn’t need to convert a sketch to JavaScript or build an instant app. It just needs to take the existing sketch and class load it instead of installing it.
With this approach, APDE only needs to build the resources with AAPT, compile the classes with ECJ, and dex the classes. On my phone, this whole process takes less than two seconds (although many phones are considerably slower), which is a huge improvement over having to build the APK and install it.
Looking even further forward, I plan to implement incremental compilation this summer, i.e. having the compiler run in the background to display the errors in real-time. With this system in place, the sketch will likely be compiled even before the user presses the run button, meaning that all that is left is dexing. In other words, class loaders could conceivably shrink the build process to less than a second, depending on the phone hardware of course.
There is much excitement yet to come this summer!
Fin
If you want to explore the changes that I’ve made in more detail, check out the android-mode-4 branch on GitHub. I have left detailed commit messages (most of the time) and all of the code is there.
Stay tuned for preview mode!