Why we moved Amaterasu to Kotlin, and what the future holds
Recently we have merged a PR to Amaterasu which removed the remains of Scala in the project (barring a few small legacy components that will surely be removed soon). After tweeting about the matter, we realized that there is a lot of interest around the reasons for said migration, so we decided to share them.
A bit of history
Originally, Amaterasu started as a Scala project. Since our main focus in the early days was on integration with Apache Spark which is a Scala framework, choosing Scala seemed to be a natural choice for us. Another reason was our familiarity with the JVM world and functional programming (I myself had Clojure background), what could possibly go wrong?
Well, as it turns out, quite a lot.
What’s wrong with Scala?
I know, this is a bit tongue in cheek. Say what you will about Scala, you probably either love it or hate it. And it is really popular (and easy) nowadays to bash Scala as a language.
But that wasn’t the case with us. We never used Scala religiously, which helped us avoid some of the pitfalls of Scala. For example, we never used some of the libraries that extend Scala to become more functional. In fact, we never even used some of the more “idiomatic” features of Scala such as some of the built in monads etc’.
That was done intentionally, as we wanted to keep an inviting and coherent code base. This also proved valuable in the readability of our code base and helped us later on in the migration to Kotlin.
In addition to that, we have decided early on to abandon Scala’s SBT in favor of Gradle, which proved as a really great choice for us.
Our issues were quite different, and in my own view had more to do with the the way Scala is released as a product rather than the language itself:
- Scala version compatibility: Scala proved to have compatibility issues between different Scala versions. This is extremely painful considering Scala is actually a runtime library and incompatible Scala versions can be present in your classpath unintentionally breaking your code. In most other projects this might not be an issue, but for Amaterasu, which runs on tools such as Hadoop YARN which in fact does tend to have it own version of Scala loaded in the classpath this was a really painful experience.
- Long build time: Another unacceptable pain for us, building the project on a clean environment was around 30 min, lot’s of fun on our CI environment.
- API compatibility: Not a deal breaker, but Scala APIs and also common Scala libraries, are happy to break APIs between versions. For me, that is just poor maintenance of a codebase that should be “production grade”. When I compare it to the lengthy discussions of other communities before breaking a language behavior (Python for example) this feels really reckless.
Why Kotlin?
We actually spent a great deal of time thinking about the right alternative. We considered going back to Java, using GO and other alternatives inside and outside of the JVM world, but after a bit of testing we found the following traits in Kotlin that really sealed the deal for us:
- JVM based and for us that meant that we can actually change gradually, no need for a big rewrite up front. this worked well for us as we managed to continue developing features, using Kotlin while slowly migrating older code. The fact that as oppose to Scala, Kotlin plays well with the rest of the JVM was also really useful. This allows us to develop our public APIs in our language of choice rather than switching back and forth to Java like we needed to back in the Scala days.
- It’s a really cool language, well designed, concise, powerful and productive, but at the same time, lowbrow unassuming and really straight forward.
- Not a big shift from Scala, in fact, in most respect it is very close to Scala, and considering the fact we had to maintain both languages for over a year this helped minimize the context switches between paradigms.
- It really took away a lot of the pain with build time coming down, compatibility issues disappearing slowly from the project.
- A great ecosystem with really useful collection of libraries such as the testing with Spek and Konf to name a few, but also language features some of which we are just starting to explore in such as coroutines and the built-in support for writing DSLs.
The next great things
Along the way, we learned a lot about Kotlin and how it can actually help us build a better Amaterasu (and a big shout-out to Eugene Petrenko and his Kotlin Deep Dive workshop and the point in the direction of DSLs). And we are planning on incorporating more and more Kotlin goodness to the project. With the main things we are looking being:
- Kotlin DSLs are one of the coolest features of Kotlin. DSLs allows projects to easily design a business language that someone who isn’t necessarily familiar with the underlying programming language can contribute.In Amaterasu’s example we have our own yaml configuration. Reading and parsing those files in runtime is not only hard with but also doesn’t give us compile-time validation and things such as IDE support.
- Better run-time performance is another thing we accept both by simply moving away from having a configuration based flow to a fully compiled process as well as by anything we saw from Kotlin so far.
- Better tests is a really key area of focus for us. Integration tests for data pipelines can, and should be better. Having a JVM based flow rather than a configuration based one really opens the door for better test flows with proper mocking of data, integrated test runners and IDE support.
Conclusion
Moving between languages was not a decision we took lightly. In fact it took us over a year to make that decision and act upon it. The main reason was that Scala was just the wrong fit for the project.
Kotlin was chosen mainly because of the apparent simplicity to migrate, but ended up really great fit for Amaterasu, and we can’t wait to bring you a more Kotlinized version soon.
If you enjoyed this, please clap 👏 so others can see it as well! Follow me on Twitter @Shintoio or Medium to get new post updates or to just say Hi 🙂
PS: Sign up for our latest product updates at shinto.io
