Is the Elixir ecosystem “mature” enough for production apps?
Have you been playing with Elixir and Phoenix? Have you been trying to decide if Phoenix would be a smart choice for your next production application? If you’re a Rails developer, or a Rails team, it’s easy to come up with a lot of reasons why switching to Phoenix might be a bad idea. In this article, I’d like to focus on just one of those reasons: “Maturity”.
“I’d love to do project X in Phoenix, but I’m worried that the ecosystem isn’t mature enough.”
This is a concern I’ve heard from quite a few Rails devs and technical leaders who are considering Phoenix. They’re excited about Elixir, love the language, and are thrilled with the power of Erlang/OTP, but Elixir is a brand new language, and Ruby and Rails represent a huge ecosystem. Consider the tens of thousands of businesses that exist to support Ruby development. The thousands of books, the training resources, hosting providers, performance monitoring companies, consultancies, massive conferences, etc, all of which exist to help you develop Rails applications. It’s inconceivable that someone would build a SaaS platform nowadays and not include a Ruby gem to consume it. Many huge and high profile websites have been built in Rails, and although they’ve sometimes had problems scaling, those problems are now extremely well-understood. Everyone knows just how to augment a rails application, how to use caching, and all the best practices for scaling out these big rails apps. It may not be a joyful experience exactly, but it’s a known and measurable problem. In other words, you can plan for it.
How can I suggest a move to Phoenix, when it means leaving all of that behind? How can I advocate such a risk? And that’s not all…
“There is a ruby gem to do almost anything I might want to do. In Elixir I’ll probably have to write a lot of that stuff myself.”
At this moment, there are more than 7100 gems hosted on Rubygems.org. The Elixir equivalent, Hex.pm has only 1455 packages. And that might even understate the case. Hex.pm is much newer, and its packages are much newer. It’s a safe bet that if we somehow came up with a way to measure the stability and completeness of the packages in each platform, we’d find that the Rubygems gems are on average more stable and complete than those on Hex.pm.
But you know you’re not going to need 7100 gems. You aren’t going to need 1455 packages either. Your big problem is you don’t actually know what you’re going to need, and that’s why that big library of Ruby gems is reassuring. It seems certain that whatever the advantages of Elixir and Phoenix, it will probably be quicker to build your as yet undefined application in Rails, because one day you’ll need to accomplish something or other, and a gem will be right there waiting for you.
But this is just a hunch, isn’t it? We haven’t really measured how much time these gems are going to save us. How could we, without knowing what gems we’ll need, whether they exist in Elixir, and if they don’t exist, how much work it’s going to be to write that functionality by hand?
Let’s frame the problem this way:
- How many gems will we need?
- How many of them have comparable packages in Elixir?
- How long will it take us, on average, to either write the Elixir equivalent of the gem we’re missing, or work around the absence of that gem in our project?
This same framing can be applied to other aspects of the Ruby ecosystem: How many [monitoring tools / SaaS integrations / hosting solutions] will we need?…
By considering the problem in this way, we can treat it as a risk management problem. If we could answer these questions, we could possibly determine what the actual cost of abandoning the Ruby ecosystem might be. The good news is that the questions are simple, objective, and unambiguous. The bad news is that we can’t actually answer any of them before trying to build our project.
As an exercise, I decided to go through all the gems used in a real-world Rails project: TRX.tv. TRX is an online exchange for the sale of television broadcast rights. The project was started a year ago, and represents many thousands of hours of work. It uses WebSockets (through Pusher), has an elaborate data model, and we’ve needed to implement many custom behaviors as we build an application that fulfills the needs of this complex business. This is not a half-assed blogging application being built as a tutorial, it’s as real-world as they come.
As you might imagine, this application includes a lot of gems, 136, in fact. But many of these gems are merely dependencies of other gems, at the top level we are “only” including 32 gems in production, and an additional 10 in development and test. I went through all of the production gems, to see which ones provided functionality unavailable in the elixir ecosystem, and the results were interesting. The gems tended to fall into one of four categories:
- Needed, and missing from the Elixir world (3) : It’s hard to say for certain of course, but my gut feeling is that the absence of these three would pessimistically represent 1–2 weeks of additional work. Spoiler: the big one is devise.
- Gems related to the asset pipeline (3, arguably 4): Phoenix doesn’t bake in the asset pipeline in the same way that rails does. It encourages you to use brunch (by default), but you can use webpack, or any other such pipeline you might desire. Apart from the training aspect, I don’t think this represents any gap in functionality. It may represent an impact in process, depending on you and your team.
- Gems less relevant, or irrelevant in Elixir/Phoenix (10, or perhaps 11): Unsurprisingly, when we shift to a new paradigm, certain tools from the old paradigm become less relevant. Gems in this category make a lot less sense when dealing with Elixir/Phoenix. As an example, Sidekiq is included in this list. With OTP as a foundation, if you’re reaching for Sidekiq, you’re doing it wrong.
- Gems we need, which DO have Elixir equivalents (15, 4 of which are very stable, 8 of which are quite stable but pre 1.0, and 3 of which are quite new): Those three brand new packages are a concern, and we’ll probably have to do a bit of work there. The most worrying of the three is the Elixir equivalent of ActiveAdmin. I’m going to call this 2–4 weeks of work, again, pessimistically.
I’ve included all the details of this analysis at the bottom, along with my justifications for putting each gem in each bucket.
If you’d like to get a real sense of whether Elixir is “mature” enough for your next project, I encourage you to do a similar analysis with one of your existing projects. If you know for sure you’re going to need to do something, you can consider it a certain and direct cost, but to see how Elixir will fit with what you tend to do on a regular basis, an analysis like this might help. Take a big existing project that represents your standard operating procedure well, and see what’s missing.
Then you should weigh that against the advantages of the Elixir platform:
- Erlang/OTP: You are not likely to find yourself in a situation where the app is working, but struggling under load, and you’re having to re-architect a bunch of things to relieve the bottlenecks. You might have to do work to scale, but it will be much simpler, and it will not require major changes to your business logic. It will most likely not require any changes at all to your business logic. I can think of a good 2–3 developer weeks that could have been saved on TRX if it had been backed by OTP. Let’s just say Pusher wasn’t the first thing we tried, and leave it at that.
- The joy of explicitness: Although there’s a learning curve up front when switching to Elixir/Phoenix, most people report that the explicitness of the language and the platform lead to a much clearer understanding of what’s going on later. Stuart Eccles’ recent article about this makes this point very well. There’s just less magic in Phoenix, and in the long run I’m certain it will reduce maintenance costs.
- The joy of writing code: Ok, this one might be a little controversial, but let me ask you: Is your favorite part of developing software seeking out and installing gems, configuring them, and then figuring out what’s wrong when they don’t play well together? I bet it isn’t. You should question whether a gem that saves you an hour or two of writing code specific to your project is actually worth the trouble.
If you made it this far, I hope you found it interesting or helpful, especially as the answer to the question I ask in the title: “Is Elixir mature enough…”, is, unfortunately, “I don’t know, it depends on you.” No one can say whether Elixir is the right choice for your team, and your project, but maybe this kind of analysis will help you answer the question for yourself. (Or for your boss.)
Finally, in case there’s any doubt. I don’t regret that TRX is built in Rails. At the time we started the project, Rails was clearly the right choice for us. I have no plans to rewrite it in Elixir, as Rails continues to be a good choice for us. Down the road though, as we start building supporting applications for TRX, there’s a very good chance that Elixir and Phoenix will be the tools we reach for.
(Thanks to Ed Ellson for reviewing my analysis, and saving me embarrassment)
Details of my analysis follow, I welcome any comments or pointers, as there’s a good chance I’ve missed tools available in the Erlang or Elixir world.
Missing from the Elixir ecosystem
rollout and rollout_ui
These gems support continuous deployment, and I wasn’t able to find anything equivalent in the Phoenix world. They enable you do separate the existence of features in an app from its deployment, by letting you turn the features on and off from the admin. We would need to write something to replace this, or we’d have to rethink our deployment strategy.
This is a simple little gem that allows you to mark ActiveRecord objects as either “read” or “unread”. I couldn’t find an elixir equivalent, although I’m not really sure how to even search for such a thing.
This was the big one. Devise is made up of many things, only some of which are available in the elixir ecosystem. Guardian and Ueberauth have you covered for authentication, and for protecting your actions, but Devise also supports forgotten password flows, email authentication of accounts, etc. There’s definitely some extra work to do here in the Elixir/Phoenix world.
Related to the asset pipeline
Several gems are related to assets. Phoenix takes a completely different approach, relying on brunch by default, but it’s optional. You’re welcome to use whatever you like. I can’t say much more on this point, as I am really not a front-end dev, but I think we can safely ignore:
coffee-rails, uglifier, and sass-rails
In addition, we use the kss gem, which is for generating styleguides. I can’t find anything in the Elixir world to support this, but there’s a node module called styleguide that looks like a possible replacement.
Irrelevant in the world of Elixir and Erlang
Several gems were included to solve specific problems encountered in the Rails ecosystem, which simply wouldn’t exist in the same way in the Elixir ecosystem, or which just don’t make sense in the new paradigm.
envyable: This allows you to set up environment variables using YAML. Elixir discourages YAML in general, preferring pure Elixir for configuration of all things. You’d write this directly in Elixir, and I don’t think it’d take very long.
dalli: This is for connecting to memcached. There is a stable elixir equivalent called memcache_client, but you’re a lot less likely to need to involve memcached in your solution, so I’ll include it in this section, but it could have gone in the first one.
exhibit: This is for working with presenters. Phoenix doesn’t work that way.
lograge: This makes rails logging less verbose. Phoenix logging is already less verbose.
pusher: For integration with the pusher service. Phoenix channels are going to work much better for you, so you’re not going to need this one.
sidekiq, sidekiq-unique-jobs, and sinatra: Sidekiq makes use of an impressive army of gems (including sinatra, for some reason) to allow you to easily use Ruby threads to spawn and manage background jobs. Here’s how you would do that in Elixir:
task = Task.async(fn -> do_long_running_thing() end)
result = Task.await(task)
There are obviously lots of other ways you might want to manage that (perhaps you don’t care about the result, etc), and they’re all equally straightforward. We definitely won’t be needing sidekiq.
will_paginate: Helpful gem for building paginators of ActiveRecord objects. There may be something similar for Elixir, but as Drew Olson has shown us here, composable Ecto queries make this very straightforward, and it’s probably not worth seeking out a plugin.
rails-12factor: This is a heroku gem that causes rails to have two behaviors, both of which Phoenix has by default. (logging goes to stdout, and the application server is capable of serving static assets)
Gems which we do need, and which are available:
Finally, there’s the gems which do actually have an Elixir equivalent. In some cases, of course, the Ruby versions of these are older, and perhaps more stable, but measuring that is even more difficult than the analysis we’ve done so far. In this section I’ve identified Elixir equivalents to rails gems, and made some attempt to judge how “stable” they are, using nothing more than my extremely fallible instincts.
Stable: I believe the following Elixir equivalents to be quite stable and robust.
pg: Postgres. We use Postgrex, which comes with Phoenix.
aasm: This is tricky, because aasm is tightly tied to Ruby’s object-oriented paradigm. But there’s a very stable :gen_fsm in Erlang, which allows you to build powerful state machines using Elixir/Erlang processes. Almost certainly a net win.
jbuilder: This is for building json. The Elixir equivalent, poison, is excellent. Very fast and stable.
newrelic_rpm: This one maybe belongs in the section above, but you probably won’t be using newrelic to monitor these solutions. Use exometer instead, as described by by Michael Schäfermeyer here. [UPDATE: Chris Bell points me at elixometer from the developers at Pinterest.]
Solidifying: These Elixir plugins are pre-1.0 release, but my impression of them is that they appear solid.
aws-sdk: For using AWS, ex_aws seems fairly complete, if a bit young.
faker: For generating fake data for tests. The elixir equivalent, also called faker, worked well for me.
mailchimp-api: for using the mailchimp service. mailchimp Pre-1.0, but seems fairly solid.
paperclip: try arc instead.
role_play: try canary.
smarter_csv: I’ve had great success with ex_csv.
elasticsearch-model, elasticsearch-rails: try tirexs.
audited-activerecord: there’s a plugin called ecto_audit, looks a bit new.
active_admin¹: There is a very new tool called ex_admin that attempts to recreate the functionality of ActiveAdmin in phoenix. There’s a lot of work to do before it gets there though, so you can count on this taking longer.
1: We don’t actually use active_admin. We have a non-OS gem that we use instead, so for the sake of our analysis, I selected a popular OS gem that does about the same thing.