5 Things We Like About Elixir Coming from Python

We’re longtime users of Python at Praekelt.org, but recently the SRE team has started using Elixir for some of our projects. This blog post covers some of our favourite features of Elixir and compares them to their equivalents in Python.

The Python and Elixir logos

Elixir is a relatively young language, having first been released in 2011. It features a Ruby-inspired syntax and runs on the Erlang virtual machine (BEAM).

1. Pattern matching

Personally, this is my number one feature of Elixir. Python does feature some very basic pattern matching, but it pales in comparison to what is possible with Elixir. This is understandable, given that Elixir is a functional programming language while Python is not.

Recently, I was writing some code to process some JSON from an API for some 3rd-party software (in this case, Marathon). That JSON had a different structure depending on the version of the software, but we wanted to be able to support multiple versions of the software.

Handling this in Python involved a lot of checking for the presence of keys and values in the JSON dict:

Because Elixir can pattern match on the structure of the JSON, we can simply write different versions of our networking_mode function depending on the structure of the JSON:

We can reach right into the JSON structure to get the values we’re interested in and we don’t have to bother with checking for the presence of keys — there isn’t a single if statement in this code.

This piece of code contains this |> syntax which leads me on to…

2. The pipe operator

The pipe operator, |>, in Elixir is a bit of syntactic sugar that feeds the result of one function into the first argument of another function. It can be thought of as similar to a Unix pipe (|). It comes in handy when applying multiple operations to some value or data.

Here’s a trivial example (and a strangely popular interview question). Write a function that, given a list of words, counts the number of anagrams (words that are composed of the same letters in a different order) in the list that are n characters in length.

In Python we might write a function like this:

In Elixir we can write the same algorithm another way:

Python has some built-in functions for performing operations on sequences of data, such as map() and filter(), and it is possible for us to write a solution using these functions. But, because these operations can’t easily be combined, it is often more readable (or “Pythonic”) to use standard for loops or list comprehensions. Elixir’s pipe operator encourages a more functional style of programming where complex operations are composed of several simpler ones.

It’s worth noting that the pipe operator isn’t only useful when working with sequences of data. It can be used anywhere that there are nested function calls. These two expressions are equivalent:

3. Immutable data structures

How many times have you written code like this in Python?:

(Yes, technically you can do a kind of obscure one-liner in Python 3.5+)

This merges the contents of two dictionaries without mutating either of the two dictionaries. This is because the update() method on dictionaries is mutating. On top of this, if one of the values (or even keys!) in one of those dictionaries is mutable, if it is mutated in the merged dictionary, that will reflect in the original dictionary. This is a very common source of bugs.

In Elixir, all data structures are immutable. If we look at the function signatures (technically, typespecs) of some common functions that operate on maps (Elixir’s equivalent of a dictionary)…

merge(map(), map()) :: map()
put(map(), key(), value()) :: map()
delete(map(), key()) :: map()

…we see that they all return a new map. It’s impossible to change an existing map because they are immutable. You can only create a new map with the changes you want.

Also, you may notice that the pipe operator is very convenient when working with data structures because many functions take a data structure as their first argument and return a new data structure.

This also eliminates data races at the data structure-level because concurrent writes (or any writes) are impossible.

4. Processes

A big reason for the adoption of Elixir on our team is its handling of concurrent code. We often build tools to manage our infrastructure. This usually requires communicating with several different services and handling events from multiple sources. This inevitably requires some level of concurrency.

Elixir uses Erlang’s actor model of concurrency using what are called processes. Each process operates independently and in isolation from any other process. State may only be shared between processes using message passing. A process can send a message to any another process that it can identify. When a process receives a message it could perform some computation, send another message, or start another process.

Do not communicate by sharing memory; instead, share memory by communicating. — Effective Go

Erlang’s processes should not be confused with operating system processes. They are actually lightweight threads (a.k.a. “green threads” — something like what gevent uses in Python-land). It is normal for thousands or even millions of Erlang processes to be running on a single system.

When we’re writing the code to take actions on a message, that code is entirely synchronous and straightforward to reason about. If we need two operations to be performed asynchronously, we split them out into separate processes.

In Python one would typically use either threading or asynchronous I/O to achieve concurrency. In the past we have written software using the Twisted asynchronous I/O library. I would argue that event-based asynchronous I/O has a steeper learning curve than the actor model. The lack of native support for asynchronous I/O in the Python language (until recently) means that most existing libraries (e.g. API clients) are written for synchronous systems and must be rewritten or retrofitted to work in an asynchronous system.

While Python 3.5 made significant improvements to the ergonomics of asynchronous code with the introduction of the async/await keywords, in the past, writing good asynchronous code has been challenging. Beyond that, because of Python’s GIL, it is difficult to achieve actual parallelism and make full use of modern, multi-core CPUs.

Every piece of software written for Elixir or Erlang was written with the actor model in mind because it is core to those languages. While there is no silver bullet to achieving seamless concurrency, Elixir starts out with a good model for achieving that goal.

5. Supervision trees

So we have all these processes running and doing their thing, but what happens when one of them (inevitably) crashes? One of the key properties of processes is:

Processes do what they are supposed to do or fail as soon as possible — Joe Armstrong in “the Erlang paper”

Well, another process has to restart the failed process. In Elixir/Erlang terms, processes that manage other processors are called supervisors. Supervisors monitor one or more child processes and restart them as needed, according to a restart strategy. The three basic strategies are:

• :one_for_one: If a child process terminates, restart that process
• :one_for_all: If a child process terminates, restart all the processes
• :rest_for_one: If a child process terminates, restart it and all subsequent processes in the list of child processes

We can choose a restart strategy based on the dependencies between child processes. Because a supervisor is just a process, it can be supervised by another supervisor. By building a hierarchy of supervisors, we create a supervision tree.

Here’s an example of a small supervision tree used in one of our projects:

Marathon Event Exporter supervision tree

The above depicts a very simple program with three (non-supervising) processes:

1. The EventCounter keeps a count of different events (i.e. it keeps the state).
2. The SSEClient connects to an event stream API and sends received events to the EventCounter.
3. The MetricsExporter is a server that can be queried for event counts which it fetches from the EventCounter.

If we consider the effect of each process if it were to stop running, we can see where the design of this supervision tree comes from:

• If the SSEClient stops, we won’t receive new events. But we will still have our existing event counts and we can still serve those. → SSEClient can be restarted independently.
• If the MetricsExporter crashes, we won’t be able to serve event counts. But we can continue receiving and counting new events. → MetricsExporter can be restarted independently.
• If the EventCounter errors out, we can’t count new events and we can’t serve event counts. → SSEClient and MetricsExporter depend on EventCounter.

Therefore, we group SSEClient and MetricsCounter under one supervisor, FrontendSupervisor, with a :one_for_one restart strategy — these processes can be restarted independently without restarting any others. We then group EventCounter and FrontendSupervisor under another supervisor, Supervisor, with a :rest_for_one strategy — when the EventCounter restarts, the other processes must be restarted.

Admittedly, supervision trees are an extra consideration for programmers on top of application logic. They are not something that a Python programmer has to think about. But they are an important Elixir primitive for building resilient applications. They force the programmer to think about the error cases in their program and what the knock-on effects of a crash could be.

Conclusion

The conclusion to this blog post isn’t that we’re never going to write any Python code ever again and will from now on write everything in Elixir.

Python is a much more mature platform than Elixir with a much larger ecosystem. Praekelt.org makes heavy use of Django and we’re in no hurry to change that. If I need to script something somewhat complicated, the first tool I’m picking up is Python, and not Elixir.

Where Elixir does shine is in the kinds of networked applications that my team builds. The above features help us build resilient tools that better handle error cases and are able to achieve the necessary concurrency.

For those interested in learning Elixir, I recommend the official getting started guide which is friendly and thorough.