Why Clojure?
In 2012, I found myself looking for a job. The startup I was working in was winding down and I spent my time idly looking for interesting things to do. Out of nowhere, Reshef Mann, whom I had worked with in a previous startup, told me that he and his childhood friend, Oren Kaniel, had just managed to raise seed money for the startup that they had been working on for the past six months — some obscure company called AppsFlyer. “What the heck?” I thought to myself, “I always liked working with Reshef and maybe this AppsFlyer thing will pan out…”
Fast forward 6.5 years and AppsFlyer is one major success story that keeps on growing. We pride ourselves on being a technology-driven company and we even took the “road less traveled” of using Clojure as the main programming language for our backend services. The interesting story is how that came to be.
Let’s get funky! Lisp, Clojure and Scala are just some of our functional programming favorites. Join us! >>Learn More
When AppsFlyer started, it was just 2 guys — Oren, the CEO, doing all the business and Reshef, the CTO, writing all the code. Reshef decided to write AppsFlyer, at the beginning, in Python. Why Python? Well, if you’re a startup that needs to grow and change code very rapidly, Python will probably serve you best. It’s a super simple language with a vast ecosystem that is both interpreted and dynamically typed, which means it’s really malleable and easy to twist and change. When I joined Reshef in writing the code, that’s what we did — we hacked away in Python to our heart’s content. We designed a microservice-oriented architecture that revolved around different Python services, which communicated mainly via pub/sub and http calls (which later evolved into a mostly async communication flow based on Kafka). This served us well for the first year and a half of AppsFlyer’s growth.
What happened then? Well, luckily for us, we became successful! We were lucky enough that some very big clients decided to give us their trust and their business. As a result, our traffic grew, and let me tell you, it didn’t double or triple itself, it grew an order of magnitude in a very short while — from hundreds of thousands a day to millions a day (and then again and again into what we call hyper growth). While this was an immensely happy period, it brought with it one major problem that every rapid-growth company faces — the old tech can’t hold the new load.
Since our backend was divided into different services, not everything failed. One simple service that served raw reports (a line-by-line breakdown of select activity by campaign and time range) started to occasionally fail due to the bulk of data it needed to process. Reshef and I evaluated the situation and we came up with these different options:
- Rewrite the service using Pypy (a more performant variation of plain Python).
- Rewrite the core of the service in C code and wrap it in Python code.
- Rewrite the service in an entirely different language/VM.
After some not-so-lengthy debate, we decided to take the third option. We knew that the business was growing rapidly and we knew that Python, for all its “goodness,” would only carry us forward so far because of inherent performance issues (Python being memory “hungry” and the GIL, which causes the process to actually be single threaded, are just two simple examples).
We also knew that if we introduced a new programming language into AppsFlyer, we would want it to be functional. Why? The simple answer is that, in functional languages, it’s usually much easier to write concurrent executing code. Since the traffic was ramping up rapidly, we knew we would need to utilize multi-threading to the max and that functional languages would make it easier for us.
Once we had that in mind, we took about 2 months to “play” with various functional programming languages. We tested Haskell, OCaml, Erlang, F# (on mono), Scala, and Clojure. At the end, we narrowed it down to either Scala or Clojure. Since both of them run on the JVM, we realized that the ecosystem there was vastly superior to the rest of the languages we had evaluated.
Scala, to us, felt like an OO language with strong functional aspects, while Clojure felt like an FP language with some OO/imperative capabilities when/if you needed them.
In the end, we decided to go with Clojure because we knew that we wanted to embrace the functional paradigm in its fullest. In Scala, a lot of times you can see plain old Java imperative code written in Scala syntax (sometimes referred to as ScaJa). That is NOT the case in Clojure since the Lisp mentality, which resonated a LOT with both of us, forces you to think and write different code.
So, after one year of working in AppsFlyer, I wrote my first Clojure service. It was kinda ugly, not very “clojury,” but it got the job done and still functioned for 3 years before we rewrote it.
Since then, Clojure has been the predominant language for our backend services. Sure, we sometimes spice it up with GO (which has actual green-threads) and Scala (for our Spark needs), but most of the time, we do it in Clojure. Today, AppsFlyer is the biggest Clojure “workshop” in Israel and we keep on getting bigger — we have tens of different services and infrastructure running on 5000 machines, all of which process around 35–40 billion daily events.
Want to write everything in Clojure? Join us
