A look under-the-hood of Simply Piano (Part 1)
How did we choose our app’s UI framework?
Simply Piano, Simply’s piano learning app, has seen a lot of success lately. This includes being repeatedly selected worldwide as App of the Day on the App Store, being chosen as one of the Best apps of 2017 in Google’s Play Store, conquering the Top Grossing Education chart in the US, appearing on the Today Show and more.
Since it’s really not your average app, and quite interesting technically, we thought of sharing some highlights and decisions about its technical stack, in the hope that others might benefit and could apply the same line of reasoning. Hence this post series.
Future parts talk about issues like how we use on-device machine learning to recognize what the users play, how we handle localization and open-sourced a framework for it, how we experiment with features and measure their impact, and more!
But lets’ start with the basics:
Part 1: How did we choose our app’s UI framework?
In the first part of this post series, I’m going to talk about the main development framework and ecosystem we chose for developing the app, and the reasons which lead us to this decision.
The thing is, our app actually consists of 2 separate areas, which are essentially very different in their UI framework.
The first part is what we call “the menus”, which as you can imagine, refers to a set of screens the user goes through before and in-between the core piano learning experience. This part is developed in the native UI frameworks for each platform: UIKit for iOS and Android SDK’s UI components. For the purpose of this post, lets’ call it Vanilla UI development.
Although the menus aren’t the part of the app in which users spend most of their session, there are actually quite a lot of screens, some of them pretty complex visually and technically, and they are probably the part we spend most of our time developing and maintaining (except content creation).
The second part is what we call the “in-game”, which is the core experience of Simply Piano. In the in-game (in the game? 🤔), you play music levels which are either a static sequence of notes or a song where notes move along with background music. All this happens while receiving instant feedback from the app if you played correctly.
The in-game provides very different technical and UI challenges, and very early on in Simply’ history, it was clear that a 2D gaming framework will be a very good solution for this part. We are currently using Sparrow in iOS, and libgdx in Android. (more on the reasoning later).
How come we’re not using a cross-platform environment?
The benefits of a common codebase for multiple platforms can be meaningful. However, we chose to go native(ish). Here’s a taste of our views of advantages of a native solution over a cross-platform one:
- A larger and more resourceful developers community. So it’s much easier to find answers or 3rd party tools.
- No performance overhead. Simply Piano has quite a few challenges in this area, and many cross-platform solutions come with a costly price on performance.
- Easy to integrate new OS features. They come out each year in Google I/O and WWDC and sometimes they can be a big win. The native development frameworks will always immediately make these features available, while with cross-platform solutions you often need to create a Pull Request if you need it quickly (or wait if they’re closed-source).
- Much better IDEs & development tools. Which often make writing the same feature twice a much faster process than writing it once in a cross-platform environment.
- We wanted to use Vanilla UI for the menus. Why is that? Reasons in the next section…
- The YAGNI principle. For reasons out of this post’s scope, our company’s strategy is to be iOS-first. Therefore, there was no reason for us to try a solution whose main benefit is easy expansion to other platforms.
The YAGNI principle proved itself so far for our first few apps, but in late 2016 we finally Needed It for Simply Piano, and we started to work on an Android version. It was clear that we need to start writing from scratch, so we wanted to revisit the original decision and see if now was a good time to consider a cross-platform solution, so at least future features could potentially be developed once and deployed to both platforms.
Our decision to have Vanilla UI menus was relevant to Android as well, so we wanted menus code to be pure Android-SDK. This immediately disqualified any cross-platform solution that makes it hard or unnatural to integrate with native UI components in the same app (Unity is a prominent example). But there were still 2 code areas to consider vanilla-UI-friendly cross-platform solutions for:
- Pure model (non UI/OS-specific) code. In theory, a well-written app that is as complex as ours has areas which are pure data structures and/or algorithms implementation, which could theoretically be written in a cross -platform environment. This is indeed relevant to our case, and we considered possible ideas like writing these chunks of code in C/C++ (which is easily accessible from Objective-C, and a bit painfully but also accessible from Java via JNI), or even using Swift for Android (Kotlin for iOS wasn’t available at the time). However, all solutions of this sort felt like they come with a substantial overhead that makes them not cost-effective.
- “in-game” code. Many of the advantages in the list above are irrelevant to in-game code when it comes to Android. Unlike iOS that has the built-in SpriteKit framework that should be awesome for our needs, there’s not really an equivalent 2D gaming framework in Android’s SDK. Writing native OpenGL code (or using some kind of other homebrewed solution over a SurfaceView) didn’t sound like fun at all, so a 3rd-party solution was required.
Eventually, the chosen framework was libgdx, which actually IS a cross-platform framework!
However, it is still written in Java, which makes it a very natural citizen in the Android ecosystem, and in Android Studio, but not so much in Xcode in iOS. There’s also a big question mark about the performance overhead it will induce in iOS for the same reason.
Anyhow, we actually implemented a WebGL-based implementation of Android as a POC in one of our ninja weeks (our version of a company Hackathon, when everyone works on a task that they choose instead of the top prioritized tasks from the backlog), and it showed great potential, so in theory there’s hope for cross-platform in-game in Simply Piano’s future.
To sum things up, apart from non-client stuff like the content (courses, songs, etc.), localization (more on that in a future post), and obviously server-side functionality like analytics or account management, our iOS and Android codebases are currently completely separate. The price is pretty obvious — it means that features that we add to our iOS app need to be rewritten in our Android app, and essentially this is the main reason why Simply Piano for Android is lagging behind iOS in features.
However — this isn’t necessarily a weakness, this is also a strength that goes hand-to-hand with our business strategy and lean-startup culture. Imagine any change in the iOS codebase had to be tested and released immediately to Android as well — This would most definitely slow us down and hinder our ability to release lean experiments and measure their impact before committing to all platforms at once.
Why are we so fixed on Vanilla UI menus?
OK, so we talked about the merits of native development in general. However, in theory, once we decided we need to use a 2D gaming engine in the app anyhow (reasons below…), we could have potentially written the entire app in the same engine, including the menus. This way also big and impressive engines like Unity become relevant again.
The reasons above all apply here: developers community, ability to take advantage of latest features, and the development tools. In addition, here are some reasons specific to UI:
Natural and OS-consistent look & feel. So, while even the menus of Simply Piano aren’t very typical for apps (for starters, we’re landscape only), they still consist of many standard UI components, like scroll views, and we want them to feel “good”!
From my personal experience, even if these components exist in the gaming engine (and that’s a big if), aspects like the bounciness at the end of a scroll view or even how a button behaves when tapped are very hard to nail and get to feel just right if not done with the “Vanilla” components.
Visual editors. Some 3rd party gaming engines also have them, but none are as powerful as the built-in visual editors in Xcode and Android Studio. As for why we prefer using visual editors over writing UI in code, although a bit dated, I think Tal Bereznitskey’s post Why Real Men Use IB lists the reasons well. If I would have to pick the top reason, it would definitely be that the UI development cycle becomes much quicker if you can have instant visual feedback on your static menu instead of having to build and run the (complex) app each time you need to see a layout change.
Easier UI Automation Testing. We are not very heavy on UI automation because we don’t see it as a very cost-effective investment. However, at least for Android, nowadays in the pre-launch report in the play console, you get free UI monkey testing which finds bugs and has saved our ass a few times. It can only fiddle with native Android components. We even wrote an Espresso instrumented test and ran it in Firebase Test Lab a few times to sanity test the app on many devices and localizations, and it was considerably easier to do this for the screens that are based on vanilla Android components and not libgdx.
This benefit is also relevant for accessibility features, like VoiceOver or Dynamic Type. I didn’t include this as a separate reason because we admittedly don’t really invest in accessibility 😔
Why 2D gaming engine for “in-game”?
While we never really tried, and while some people suggested it when we consulted with them about the concept, our intuition is that a “Vanilla UI” based solution wouldn’t be comfortable enough and quick enough for the in-game’s mission: rendering complex notation with fancy animations while playing background music and running machine learning models on device to recognize what the user is playing. All this in 60 frames per second.
The in-game also doesn’t really earn much from the benefits of Vanilla UI like the menus do, as the UI is very dynamic and doesn’t include many buttons or standard components.
On the other hand, the benefits from a 2D gaming engine are substantial. These engines make it very straightforward to run a 60fps game loop, keep a complex 2D scene and manipulate it in real-time, run custom animations, pass events between different game components, etc.
One of the cooler concepts of such frameworks is Texture Atlases. They let us load many graphical assets and PNG animations to memory while making it very effective for the GPU to render all of them in one draw call, which can really benefit the performance. Here’s a frame capture from the app that shows how some of its draw calls look in Xcode’s GPU-capture tool, which is made possible thanks to the use of Texture Atlases:
How did we choose which 2D gaming engine to use?
The reasoning of using libgdx for Android was already addressed in the section about cross-platform environment above. iOS is a bit of a different story.
Going back in time to 2011, with the decision to be iOS only for our first piano app (Piano Dust Buster), we were looking for a 2D gaming solution to go hand in hand with it. Back in the day, there was no SpriteKit, which would definitely be a top option we would consider if we had to make this decision today. One of the top options back then was Cocos2D, which eventually evolved into cocos2d-x, a popular cross-platform engine in C++. However, we eventually went with what seemed like the #2 choice at the time, the Sparrow framework. The reason in retrospect is a bit stupid I must admit: we had a Recorder app written in ActionScript3, and Sparrow’s components were highly inspired by ActionScript’s display components, so the reasoning was that porting relevant parts from the Recorder app will be easier and more natural. Eventually, we didn’t end up doing anything of this sort, so this reasoning proved irrelevant.
In any case, I’m pretty proud to say that the in-game code of Piano Dust Buster was written to the highest of standards, including unit tests and modular decoupled components. This makes much of the original code still relevant and still in-use in Simply Piano, and the cost of porting it to a different 2D gaming engine today is quite high. So we stayed with Sparrow till this day.
Unfortunately, since SpriteKit came out, Sparrow practically became a legacy framework. An example of how it was problematic came when Apple asked us if we can incorporate 3D Touch into our on-screen piano in Touch Courses. Since the piano was implemented based Sparrow’s touch system, which didn’t add support for 3D touch, we eventually had to implement it ourselves in a fork of Sparrow and create a Pull Request. So I would definitely not recommend Sparrow for a new project — consider using SpriteKit instead.
Summary
I hope you found the technical details and reasoning behind them interesting. I’m sure that if we would have started today from scratch or if it was someone else making the decision we could end up with a very different framework selection, and maybe it would fit just as well or even better, but as everything else in software development — there’s no one right way.
I would be super happy to hear your feedback and answer any questions you might have in the comments. If you found this post interesting, be sure to read Part II:
Also, if you feel like working on future versions of this app or other products by Simply, and be part of the team that will affect future decisions such as these, be sure to check our careers page :)
