The benefits of using Elixir for backend development: Our perspective
Some time ago we encountered a production issue: One of our Elixir applications suddenly started responding intermittently. After a quick analysis of what could be causing it, we identified abrupt and excessive RAM consumption linked to weak CPU consumption, which led the application to stop responding for a few seconds at a time.
Beyond the incident itself, this story is interesting as it is a perfect example of the resilience of an application written in Elixir. If a problem occurs, an Elixir application is capable of restarting some of its own functions and running again within a few seconds. This is a particularly significant feature for products that have many users and/or eager features, as is the case at Welcome to the Jungle (WTTJ).
At WTTJ, we have been using Elixir in production for five years now, and in this article we are going to share our feedback on how things have worked out.
From Ruby to Elixir
WTTJ’s story did not begin with Elixir. Our first two minimum viable products (MVPs) were written using the Ruby on Rails framework. At the time this allowed us to work faster, thanks to its easy-to-use design and the framework’s scaffolds. For the first two years, this stack performed extremely well: We had two functional products that we could evolve rapidly.
But since Ruby on Rails is based on Ruby — an interpreted, single-core language — our app was a monolith that was difficult to scale horizontally. And with the number of users increasing yearly, we were experiencing more and more issues with response times and processing failures. After several attempts to optimize the code and libraries, we quickly reached the language’s limits.
By the mid-2000s, multi-core processors had taken over the market and it was time for us to find a more suitable programming language that could be both multithreaded and multiprocessing to address our growth challenges.
We therefore put several languages to the test, mainly compiled languages with multi-core support: Crystal, Go, and Elixir. Despite its promises, Crystal was a new and unstable language and we simply did not want to risk managing our main product with technology that still needed work itself. As for Go, it was also quickly ruled out as our teams were not fans of the language’s paradigms. Furthermore, feature-development times would have been greatly extended without a web framework that matched Ruby on Rails’ standards. We therefore chose Elixir, which WTTJ has been using in production since September 2018.
Now let’s just dive straight in with our feedback on the language’s ecosystem, the elements relating to the development itself, and everything relating to deployments.
The Elixir ecosystem
The Phoenix web framework that comes with Elixir is undoubtedly one of the ecosystem’s major assets and what led to the language’s approval within our teams. Even though Elixir was still new when we started using it, we could easily create web applications using the framework, thanks to its paradigms being similar to Ruby on Rails and the possibility of natively using WebSockets, a popular technology at WTTJ. It has to be noted that we use this framework mainly for its API capabilities and our use of the Views and LiveView systems is quite rare.
It is also worth mentioning that, although this stack is elegant and efficient, Elixir is still a niche technology. According to the StackOverflow study published in spring 2022, only 2.46% of professional developers say they have used it or would like to use it soon, despite it being the second-most-loved language (what a paradox!). This results in strong limitations for companies using this language.
First of all, this niche positioning leads to recruitment challenges: Finding developers with experience in Elixir is complicated. It is therefore often necessary to consider recruiting developers interested in the language but with no significant experience, and then dedicate time to training them. This incurs additional company costs, since these developers will obviously not be able to meet their objectives for several months.
The other drawback is that the community is small, of course. This means that the language is rarely integrated, service providers do not always offer SDKs in the language, and fewer libraries exist so there is less competition for some features.
That said, this does not make Elixir’s community any less welcoming — in fact, it’s quite the opposite! As contributions are rarer, proposing new projects or porting existing features is highly appreciated and easier to be put forth by the community. In 2022, we were delighted to be invited by the Thinking Elixir podcast to talk about our ecto_anon project and our articles also regularly feature in the ElixirStatus newsletter.
Technical insights
The primary benefit of Elixir for development is that the language is functional, which means it is mostly declarative and its variables are immutable. Moreover, the language’s creator, José Valim, a former core developer on the Ruby on Rails project, has given Elixir a “Ruby-like” syntax. He has made it possible to implement several notable features, such as pattern matching — a syntax allowing several functions to be written with the same name but different arguments — and weak typing (weaker than Go or Crystal), which makes it easier to deal with external APIs, for example.
Ruby on Rails’ influence has also given Elixir the logic of Active Record, an object-relational mapping (ORM) system. As we understand it, Elixir looked to Active Record when creating the SQL wrapper Ecto, which focuses on explicit queries, even if the code is verbose. But whereas Active Record only lets you write queries in object-oriented language, meaning you lose potential SQL optimizations, Ecto allows you to write SQL as if it were object-oriented.
The Elixir language also facilitates the use of test-driven development (TDD) because of the way the code is compiled. If there is an earlier compilation in Elixir, the compiler simply recompiles what has changed, and this benefit is proportional to the application’s size. With some other languages, if you compile a substantial application from scratch, you have to run multiple compilations for each tiny code change. If your code is well structured, consecutive test runs can be conducted at incredible speed.
Another asset of the Elixir ecosystem is Plug, which you will most likely use if you work on web applications. This awesome library allows you to easily use tools or libraries in your project, such as integrating the Cowboy web server in two lines of code. But it also allows you to link all your adapters, such as your custom application router or the web session management system.
However, Elixir also comes with its fair share of challenges. One of the issues we first encountered involved the management of configuration variables. Many things are set in the release at compile-time, whereas we previously had on-demand dynamic environments at WTTJ. We therefore had to run the same release with different configuration variables. It should be noted, however, that the tools for managing compile-time vs. runtime variables have evolved since then and this differentiation is now easier.
The rake task launching procedure was also impacted by the move to Elixir and Phoenix. The language offers the possibility of launching mix tasks with Mix, but these tasks are only available in the development environment, since Mix is not a tool intended for production. It was therefore necessary to set up subterfuges in order to be able to execute certain migration tasks in production.
The releasing process
When creating a release, some rules must be followed. One of the most fixed rules is that the compilation and execution environments must be ad hoc: The code must be compiled using the same processor family, the same distribution (using the same family of distributions, e.g. Ubuntu and Debian, is not enough), and the same version. This has many implications in the process of setting up the delivery process. When you update your production environment, you first have to update, and of course test, your integration process. However, Elixir has an unquestionable benefit: It allows you to embed its ecosystem in it — there’s no need to install Erlang or Elixir on the destination machine! While this language feature generates heavier releases of several megabytes, it also crucially limits friction in the delivery process.
Moreover, since Elixir is based on the Erlang virtual machine, a language initially dedicated to telecommunication, one of the advantages is that applications written in Elixir work very well in clusters. Protocols exist for them to communicate easily and thus improve overall resilience.
To do this Elixir relies on GenServers (for generic server processes), which are processes internal to the application that allow states and behaviors to be isolated. The main strength of these GenServers is that they bring a high degree of flexibility to an application, particularly with the use of the supervision tree, which makes it possible to be fault-tolerant by isolating faulty processes without bringing down the whole application. Remember our application mentioned at the start of this article that stopped responding for a few seconds? This was due to a GenServer restarting.
There are also several distributed storage systems in the OTP (Open Telecom Platform) framework, such as ETS, DETS, and Mnesia. We have used Mnesia, a distributed real-time database management system (DBMS), in some projects at WTTJ to speed up and facilitate access to fast information within a cluster.
However, the system for deploying Elixir applications can be complicated to use for beginners. When we first started using the language, we faced difficulties with Distillery (which is now obsolete), exacerbated by the lack of resources available at the time. Beyond the complexities of compile vs. run environments, back then Distillery was an early version of a release system probably not adapted to the web. But it is worth noting that the language and its ecosystem have since made significant progress in this and we’re now deploying our applications using mix release, which is built in.
You won’t regret it!
So, do we regret choosing Elixir as our main backend language? Absolutely not! The language has proven to be very stable despite it being relatively new, and all in all, the learning curve has been rather similar to that of any other functional language. The Elixir universe is denser than you first expect, so it is worth taking the time to fully understand the Erlang/OTP ecosystem when starting out. But as you can tell from our experience, it is definitely worth it! Have we convinced you? If you want to take things a step further, check out the Exercism website, which offers a free, high-quality training course on how to code with Elixir.
Written by Charles Petit, Head of SRE @ WTTJ
Thanks to Kevin Lacointe and Stéphane Robino for their valuable contributions to this article
Edited by Anne-Laure Civeyrac
Illustration by Myriam Wares
