3 Months with Elixir & Phoenix
In the career arc of the modern software engineer, rising from the ashes is a common theme.
One of the joys of building a brand new product is the opportunity to take a step back from the tools one has been using and evaluate new technologies and frameworks. This doesn’t mean switching tooling for the sake of something newer and sexier — it means being thoughtful about ones experiences and looking at new & better ways to solve the problems one invariably encounters. So when we started to build out the Verifiablee app a couple of months ago, I decided it was the perfect opportunity to try out some new tooling I’d had my eyes on.
Rails Battle Scars
Before going any further, I should probably clarify what kind of app we’re talking about here — different apps obviously have different needs. We have a couple of different services in our product, but the app I’m writing about now is a pretty typical modern web-app, with all the baggage that entails. That means that the core functionality is serving up web content with CRUD APIs, but also means things like 3rd party integrations, authentication, background jobs, asset pipelines, etc.
Because we were building a brand new product and wanted to bring it to market quickly, developer productivity was one of my biggest concerns. Having built many different products with Rails over the years, and because of its emphasis on developer productivity, it certainly would have been a reasonable choice for this app. But, like most other seasoned Rails developers, I’m sporting my share of Rails battle scars, and was eager to explore alternatives.
There are of course lots of things I like about working with Rails — otherwise I would have moved on long ago. In particular, I appreciate the focus on developer productivity and the ecosystem support. Many of the design decisions around Rails are oriented towards productivity and building products quickly, and this remains the framework’s biggest strengths in my opinion. On the other hand, like many others, I’ve witnessed the kinds of issues that can arise in Rails projects as they scale both in complexity and in usage. Rails apps for example, are fairly bloated from the start with respect to startup, runtime and memory performance, and these problems only grow as projects become more complex. Additionally, the dynamic free-for-all approach of the Ruby language, with it lack of static typing and other compile-time checks (which enables much of the Rails “magic”) can also lead to difficult-to-maintain code as codebases grow.
So while I love a lot of the design decisions and the general philosophy behind Rails, working on a growing Rails project can become frustrating quickly.
Enter the Phoenix
What if there was a technology that took all of the good ideas from Ruby & Rails but fixed all the bad stuff, like the bloat & the performance, and put it on a more solid foundation? It turns out that’s essentially what you get with the Phoenix Framework on Elixir, the language built by José Valim, a long-time contributor of both Ruby and Rails.
I’ve had my eyes on Elixir for a couple of years and have used it in some side projects, but I had yet to fully immerse myself. Side projects and tutorials are great ways to get exposure to a new technology, but it’s not until you really dive into building a shipping product that you really get to know and understand it. Personally, I need to run into all those real world problems that are easy to overlook when taking a new framework for a casual spin: what’s the tooling like, what’s the ecosystem like, how the heck do I deploy this thing?
Using a new technology in a live production environment always reveals new challenges and opportunities to learn, and while I certainly won’t call myself an expert on Phoenix & Elixir, building this product has given me more than enough experience to tackle some of those challenges. I’ve captured some of my thoughts and leanings here. This isn’t meant to be an exhaustive analysis of Elixir or Phoenix, but instead, some of the key observations I made in building out a new product.
Pure Functional Programming — Mostly
The first big win for Elixir is the focus on functional programming. I’m a big fan of functional programming in general: managing and reasoning about program state is often difficult and can be the source of insidious bugs. I’ve always appreciated that Ruby allows and often encourages developers to use a more functional programming style, though it does not force them to do so. At the micro level, using functions like map and reduce are certainly more idiomatic Ruby than writing for-loops, but at a more macro level, Ruby’s object-oriented nature tends to dominate, with state stored at the class and instance level.
The Elixir language is, at its core, entirely functional (though programs do maintain and manipulate state via processes in the Erlang runtime — more on that below). By removing state, code is simpler to reason about, debug and test, though it does occasionally require bending ones mind to solve a problem that may be more easily accomplished in a non-functional fashion.
So what about this state management via the Erlang runtime? The Erlang runtime allows for concurrent processes (which are not true OS processes and more analogous to threads in other languages), and these processes can, in effect, have their own fully isolated state. Applications can thus make use of state by sending messages to other processes. These messages are processed in the order in which they are received and may perform other computations than just changing state, such as data validation and other forms of state management.
Furthermore, these messages are typically not specifying state directly by saying “hey process, set variable X”. Though that can occur in some use cases, such as caching, the messages are more often higher-level application semantics like “hey process, the user made this chess move, update the game state accordingly and send it back to me”. This encourages data transformation and validation built-in to the state management. In a sense, manipulating state in Elixir can be thought of a little bit more like making an extremely lightweight API call than like setting a variable.
The reality is that sooner or later, most useful applications need some notion of state, it’s just a question of where and how languages expose this functionality. By avoiding state in the core language and by isolating and abstracting it into separate Erlang processes, Elixir manages to drastically reduce the use of state and mitigate many of the pitfalls that can come along with stateful solutions in other contexts: data inconsistencies, deadlocks, thread contention and race conditions.
Metaprogramming & “Magic”
Metaprogramming in Ruby, and its application in Rails, is a contentious issue. The use of metaprogramming in Rails is often referred to, both complementary and disparagingly as “magic”. It is used to enable many of the benefits of the Rails framework; and is abused to create many of the downsides. Chief among those downsides are that code that is opaque and difficult to understand, as well as code exhibiting unintended consequences and side-effects.
Unfortunately, this is one area where in my opinion, Elixir carries at least some of the same baggage as Ruby. Though the use of metaprogramming is somewhat more restrained in Phoenix, Elixir actually enables far more metaprogramming functionality than Ruby via its powerful macro system. Elixir macros allow developers to effectively re-interpret the core syntax of the language and transform the meaning of expressions. It’s pretty much as scary as it sounds, though as the documentation warns:
Elixir already provides mechanisms to write your everyday code in a simple and readable fashion by using its data structures and functions. Macros should only be used as a last resort. Remember that explicit is better than implicit. Clear code is better than concise code.
For an example of how macros are (ab)used, let’s take a look at Ecto, the Elixir analog to ActiveRecord. In ActiveRecord, I might write a query to fetch the names of users over a certain age like this.
User.where(:name => query).select(:name)
In Ecto, using the preferred query syntax is a DSL implemented via a macro:
from u in User, where: u.name = ^query, select: u.name
It looks reasonable, but there are several features of this line of code that do things completely different from what they normally do in the Elixir language.
- “x in y” usually tells you if the element “x” is in the Enum “y”. But not here.
- “^” is usually the “pin” operator used in a match clause, which is kind of conceptually how it’s being used here, though we’re not in a match clause, so it would not actually be valid (or expected!)
- “where: x, select: y” would usually be creating a keyword list using the values for “x” and “y” — but that’s not happening here either (because the binding of “u” at that point is not what we would expect)
Now having read the documentation for this particular, commonly used method, I get it and I understand how it works. And because under the hood here this method is essentially assembling a SQL fragment, some form of code gymnastics is going to be necessary. But in my opinion, these types of macros generally make code more difficult to read & reason about.
I’ve found that this type of macro strategy is used in a surprising number of Elixir libraries. Any library which is integrated using the Elixir “use” keyword can inject unseen code into a module which may (or may not) make use of this kind of metaprogramming.
I understand and appreciate the need for this kind of functionality in the core language, but am sometimes taken aback by its applications, especially in a language which puts an emphasis on explicit behaviors, though as detailed in this thread, there are varying opinions on what explicit really means.
Developer experience & ecosystem support
One of the biggest benefits of Rails in my opinion, is the rich ecosystem: it’s been around for long enough that somebody, somewhere has most likely already solved the problem you’re trying to solve — and there’s probably a gem or a blog post to help you solve it yourself.
I’ve been pleasantly surprised to find that Phoenix, while being relatively young, also has a healthy ecosystem and packages available for most of the integrations and other features I’ve needed. Sometimes, while Ruby might have 3 or 4 different gems to choose from (2 or 3 of which are likely outdated and unmaintained), Elixir might only have a single package available. But in my experience that package is more likely to be up to date and functional. It would be hard to deny that Rails holds an advantage here, given how long it’s been around, but in practice, it hasn’t been an issue for any of the problems I’ve come across.
In spite of being much younger, the general developer experience of Phoenix easily matches or surpasses that of Rails. Phoenix, like Rails, takes a mostly batteries-included approach, meaning that there’s a focus on developer productivity and having core functionality right out of the box. There are of course differences in exactly which batteries are included, so to speak, but generally speaking both frameworks deliver on this promise.
One feature of Phoenix worth calling out separately is one that I’m not using extensively right now is LiveView, which is a server-side rendering solution built into Phoenix out of the box. LiveView allows for real-time, server-rendered, single-page apps without writing a line of JavaScript. Building the full app in LiveView was a bridge too far for me (I went with React for the client), but there are a couple of specific features I’m working on that seem like ideal match for LiveView — situations where there’s an interactive component combined with a lots of back-and-forth between the server and the browser. I’ll be evaluating LiveView for these use cases as I build them out further.
OTP & the Erlang runtime
The most fascinating aspect of Elixir is also the one where I’ve had the least experience: the Erlang runtime with Open Telecom Protocol (OTP). It’s probably more accurate to say this is the area where I have the least explicit experience — if you’re writing an app in Elixir or Phoenix, you’re implicitly experiencing the benefits of the OTP from the start.
The Erlang runtime is more complex than most language runtimes and is almost like a mini operating system for apps running on the Erlang VM. It’s designed for massive concurrency and fault tolerance and promotes a “let it crash” philosophy, in which individual processes are allowed to crash or fail without bringing down the entire application. Hierarchical supervisor trees are used to monitor processes and gracefully handle any crashes that occur.
What does this all mean in the context of Phoenix, for example? An “app” written with Phoenix is typically made up of dozens or hundreds of individual processes, most of which are not explicitly started by the developer. Each web request to the app is its own process, as are lots of other pieces of functionality behind the scenes: websockets, database connections, connection pools, loggers, and much much more. Even if you never explicitly create a process, your Phoenix app is taking advantage of the OTP for stability, scalability & complete isolation of all the moving parts of your app. The OTP is a clear win for Phoenix over other frameworks like Rails.
The recent release of the Phoenix Live Dashboard gives a small glimpse into the OTP and how it’s used in a Phoenix app. This drop-in lightweight dashboard shows process monitoring and metrics that demonstrate what the OTP does in every Phoenix app.
3 Months Later
After 3 months immersed in Elixir & Phoenix, I’m very happy to have made the leap. As I mentioned at the start, my use case is mostly a vanilla web app that has relied on the support of excellent Phoenix framework, and hasn’t required many of the more exotic features of the OTP — so there are many topics I’ve yet to fully explore. I know there are a great deal of problems that I haven’t yet had the need to solve, and surely sooner or later I’ll come across one that will leave me banging my head on the keyboard — as will occur with any language or framework.
That said, I consider the choice of Elixir and Phoenix to be a great success. In fact, I’ve raved to anybody who has asked that Elixir is the “language of the future”, with only slight hyperbole — there are a lot of great new languages for application development these days and I haven’t done a deep dive with all of them of course. But after spending time working with the language and in particular the OTP, I do feel safe saying: there is something very special about Elixir.
