Our experience writing Elixir microservices
Elixir is usually chosen by developers around the world to develop monolithic backend applications (especially when combined with the Phoenix framework). So, after using it successfully for more than a year in a microservices environment at Heetch, we wanted to share our unusual experience!
How did we end up here?
In 2015, after spending 2 years scaling our backend team around a monolithic Rails app, we started to experiment with other tech stacks. We quickly started to plan a microservices extraction, since that seemed to be the best choice for scaling both the team and the app.
We took a look at Elixir but, at the time, we found that with our use case we would have to reach for Erlang more often than we were comfortable with, and hiring experienced developers could be a challenge.
After considering our options, we picked the Go language to work on extracting parts of our monolith. The first months were spent learning about challenging topics such as orchestration, instrumentation, or inter-services communication (all of which are worth a blog post by themselves!), but we ultimately ended up in 2016 with around 20 Go microservices successfully running in production.
Combined with a reorganization of the engineering department into product teams, a microservices architecture proved useful. But as mainly Ruby developers, some of us kept wondering whether Go would be a fit for every team and in every use case.
Around the middle of 2017, we took another look at Elixir and, impressed by how far the ecosystem and community had progressed, we decided to give it a serious try!
Our elixir microservices stack today
When we started experimenting with Elixir, we made a pragmatic decision to NOT fully embrace some of the more advanced features of the OTP platform. OTP is one of the selling points of Elixir, and it allows you to run an entire system as an always-up single app, distributed within a cluster. However, we already standardized our process around a more classic microservices architecture, with small applications that are small and simple enough to be supervised by our infrastructure tools, and we did not want to stray too far from it.
In practice, we are building applications that are deployed as Docker images. Our apps are small and not interconnected within an Erlang cluster, but integrate within our microservices mesh already composed of existing Go and Ruby services. This helped us stay in familiar terrain while we got our feet wet.
This unconventional approach of running Elixir still allows us to benefit from the many advantages the language and ecosystem has to offer, so let’s take a look at some of those points in details!
Application architecture: At Heetch, our microservices communicate between each other via HTTP endpoints speaking JSON, and asynchronous events sent via Kafka.
Since all of our services are standard Elixir applications, Mix, Elixir’s build tool, abstracts most of the boilerplate and developers that are unfamiliar with the codebase can easily discover the entry points of a service by looking at its application definition:
defmodule MyService.Application do
def start() do
# List all child processes to be supervised
children =
[
sql_database_connection()
redis_client(),
phoenix_http_endpoint(),
kafka_consumer()
] Supervisor.start_link(children)
end
end
This application definition creates and configures a supervision tree, which plays a huge part in the scalability and fault-tolerance capabilities of Erlang/OTP.
Phoenix: Phoenix is often spoken of as a replacement for heavy frameworks such as Ruby on Rails or Django. But contrary to most MVC frameworks that enforce their own structure, Phoenix applications are nothing more than standard Elixir applications! Boilerplate code can also be reduced to a minimum, so that we only need to set up a couple of files to start.
The controller and views system proved useful even in our use case of a JSON API, and with pure Elixir templates files we do not need per-model serializers which can sometimes become hard to maintain as they get reused across endpoints.
As a result, we use for Phoenix in most of our services, even when our needs are limited.
We also recently had a successful experiment implementing a Graphql API using Absinthe for our Driver Portal (a React-powered website where drivers can manage their accounts). Absinthe helped us build a “driver profile” microservice that elegantly stitches together data from multiple internal services in a single schema, improving the lives of our frontend team that previously had to deal with many different RESTful endpoints.
Ecto is a « toolkit for data mapping and language integrated query for Elixir », a fancy name for the most popular database querying/data mapping library and what may be a “killer feature” of Elixir.
PostgreSQL is go-to choice for storing data (keeping our services stateless), a lot of our microservices use Ecto and we are delighted by the SQL query builder, but also by features such as the schemas and changesets which we use as building blocks in our business logic.
Even though we reach for Redis when we need a cache that is consistent between multiple instances of a service, we also make use of the built-in Erlang Term Storage engine for storing ephemeral data, via the con_cache library.
As you’ve noticed we mainly talked about the OTP framework and the various libraries until now. We could keep talking about the excellent Mix tool that handles compilation, project configuration, dependencies management and contributes to a development experience that feels polished and, like Ruby, « optimized for the programmer’s happiness ».
But let’s not ignore the language itself! Elixir wraps functional programming concepts such as immutability and pure functions in a friendly syntax and feels extremely expressive.
If you haven’t tried Elixir yourself yet but discussed with users of the language, they probably talked your ear off about features such as pattern matching. Simple examples may not fully showcase its usefulness, but we enjoyed applying pattern matching in various areas of our code such as our Kafka consumers:
# handle_message is called every time we receive
# an event from Kafka.
def handle_message(%KafkaMessage{
payload: %{
"event_name" => "user_created",
"user" => user_data
}
}) do
initialize_user_account(user_data)
end
def handle_message(%KafkaMessage{
payload: %{
"event_name" => "user_deleted",
"user" => %{"id" => user_id}
}
}) do
close_user_account(user_id)
endUnfortunately, not everything is perfect, and we hit a few roadblocks in some areas such as (surprisingly!) deployment.
The standard way of running Elixir or Erlang applications is by using releases, which bundle your application, dependencies, and the Erlang VM together so that you can run your code by executing a single script.
As mentioned earlier, our services are deployed as Docker images, and we apply the «12 factor» philosophy of using environment variables for configuration. This allows us to use the same images for both staging and production. However, our releases did not play well with this concept, since libraries like the standard logger are configured using files that are provided during compilation.
As a workaround, we currently do not use releases but instead directly include elixir in our Docker images. This allows us to put code that fetches environment variables in our config files, and run mix tasks similarly to how we do during development, but results in bigger images.
Note that the distillery project (in its 2.0 version) brings tooling that helps solve issues related to releases. We may adopt it to make our build and deployment process a bit more elegant than today!
A success so far
After almost 2 years using Elixir, we currently have 2 product teams using it daily and maintaining more than 30 services.
Thanks to our internal « Developer Care » team, we have an internal framework that abstracts most of the boilerplate required for common concerns such as logging, metrics or request tracing, by providing libraries and plug middlewares.
Overall, developers are happy and productive, and whenever we need to work on areas of the product that are still handled in our Rails legacy monolith, we take some time to extract the code into a new elixir microservice.
More than 3 years ago, José Valim, the creator of Elixir, published an article about Elixir and its place in a “time of microservices” that argued that your backend can be written as a single modular Elixir app, but we humbly hope to have shown with this article that you can also adopt Elixir as part of a larger polyglot codebase. If you have a similar experience, we’d love for you to share your feedback on this topic!
