Two Years of Elixir at Podium: Thoughts

Exactly two years ago today, our largest Elixir app hit production. We had two other, smaller, low-risk Elixir apps running in production at the time. However, with this larger app, once it hit production, there was no going back. We named it “Black Mamba”, after the world’s fastest snake. We had high hopes for this app.

It’s been two years since now, and we can honestly say in our heart of hearts that we have had absolutely no remorse or desire to switch languages. Of course there’s talk every once in a while of writing some very performance-sensitive code in Rust, or maybe seeing what Golang is all about, but we’re all in on Elixir, and we couldn’t be happier about it.

First, a little about our code ecosystem. We have over 110 Elixir repos. Of those, 11 are public libraries, a small handful are private libraries (usually to simplify communication between microservices), and the rest are all server applications, all written and managed by 80 (and growing!) engineers, split across 17 teams. Basically, we have a lot of Elixir code, and a lot of engineers working on Elixir code. Now that you have some context, here are some of our thoughts on Elixir:

It’s really easy to pick up

Most people we hire have no Elixir experience. Many have some Ruby experience, which helps with syntax, but we’ve noticed that for our new hires, picking up Elixir is almost never an issue. It might take a little while to really get a grasp on processes, genserver, macros, etc…, but most of our new hires get code pushed into production on the first day. The syntax similarities with Ruby help somewhat, but Elixir’s functional programming style is what really makes it easy to jump into a codebase and make changes, even having never worked with the language before. In our experience, it is simply much easier to reason about.

There are some things that are a little tricky, of course, especially when compared with Ruby/Rails. Ruby’s Active Record makes it very simple to interface with a database, while Ecto can be a bit intimidating. But as you start needing more complicated queries and start adding callbacks, Ecto comes out ahead. Ecto is a lot closer to raw SQL than Active Record is (which is part of the point), so where adding something to an Active Record query requires you to drop down to SQL strings, in Ecto you can still build up the query using the standard DSL.

It’s reliable

Podium handles the interactions between local businesses and their customers. Necessarily, we integrate with a lot of third-party services. Since we try to handle the most commonly used services, most of the services we integrate with are the de facto “systems of record” for a particular industry. Services like these don’t have much competition. Let’s just say… sometimes they can be unreliable or difficult to integrate with. (We’ve gotten quite good with SOAP APIs). Our first integrations were all run through a single node app, which we called “integration-hub” (this is one of the few repos of ours that has a meaningful name). Comparing this app with our current Elixir integration apps, our team lead over integrations said that the major downfall of the NodeJS app was that a single poorly handled error for one of the integrations would tank the whole application, requiring an engineer to look into the issue for a remedy. Due to the flaky nature and ever-evolving apis of integrations with third parties, this was a huge issue. On the other hand, with Elixir, we can isolate responsibilities to different process supervisors and restart child processes if something goes wrong.

Everyone else’s docs make us sad now

I’m just going to say it: Elixir has the best docs. A lot of libraries in other languages will go and build themselves a nice documentation page, but nowhere is it as standard as it is with Elixir. I rarely use Google or Stack Overflow to solve a problem anymore; another engineer showed me that I can just go straight to the docs. The answer is almost always there, and if it’s not, it usually just takes a quick glance at the source code. It’s not just the fact the Hex docs are standardized, it’s that most libraries do a very good job of documenting their code. Elixir, Plug, Phoenix, and all of the very commonly used libraries provide amazing examples of documenting code. I wouldn’t be surprised if there were more comments than actual lines of code in these projects.

The community is pretty great

If you go to any random Stack Overflow question, Elixir Forum post, Github issue, or Hacker News comments section, chances are you will find a core contributor to Elixir or Phoenix (we have a theory that José Valim, the original author of Elixir might actually be several people based on his prolific Stack Overflow answer volume). Even when the person giving help is not a core contributor, they are usually very knowledgeable and helpful, and most of the big libraries have their own Slack channels, where the authors of those libraries (or again, just people that have some experience with it) will help you out with any question. The size and age of the community and language is often cited as a possible downside of using Elixir, but we haven’t really run into many issues related to that. Occasionally, there might be some missing libraries for some API, but the community is so involved that it makes up for its smallness. Is it hard to find Elixir developers? Sure, but due to the rock-solid nature of the language our new hires pick it up very quickly.

It’s really fast

Our very first Elixir app (one of the aforementioned smaller apps) was a link-shortening microservice. Fun fact: we named it “Barbados,” after the Barbados Threadsnake, the smallest snake in the world. We’ve named a lot of different repos after a lot of different snakes for some reason, but that’s a blog post for a different time. It was written over the course of a single a weekend, and on Monday the engineers that built it couldn’t wait to show us that the response times were 20μs. We knew it was going to be faster than our Rails monolith, but we had no idea it’d be sub-millisecond. To be fair that app doesn’t really do much beyond parse an id out of a url, look up that id in a database, and then return a redirect to the full url, but still, that’s fast. And this app’s endpoint probably has the highest throughput out of any of our endpoints. There are faster languages out there, there always are, but in addition to the response time, we’ve been running this app for almost three years, have only pushed 8 meaningful commits, and I don’t know if it’s ever gone down. I’m sure we’ll eventually run into performance issues as we grow, but at this point most of our bottlenecks have been due to problems like unoptimized database queries; we’ve rarely had to refactor Elixir code due to performance reasons.

You have to be deliberate about your data structures

Elixir’s core set of data structures are fantastic. However, if you’re coming from a JavaScript background, they take a little getting used to. Almost all of our engineers are full stack, so they usually have a good amount of JavaScript experience. In JavaScript, you really only have one data structure, the “object” (arrays are just objects with integer keys). Elixir has maps, lists, structs, keyword lists, tuples, and others. It can be tricky to know when to use a map vs a set, or a map vs a keyword list. It’s not exactly clear-cut, and some of it has to do more with convention than anything else. One of the biggest issues I see developers face is forgetting that a list is not an array. We don’t waste too much time trying to figure out what the Big O of a function is, and generally it’s an issue of multiple O(n) accesses and can be solved by a single Map.new/2, but it’s still a consideration we don’t have to make in Ruby or Javascript.

In Conclusion

When that first major app went live, it had 6,500 lines of code spread across 109 files. Two years later, there are 93,000 lines of code across 950 files. There are about a dozen developers deploying code several times per day, and there’s just something about it so that that’s not an issue. We had to put very strict code review and testing policies around our Rails app because people kept breaking it, and while that’s not all Rails’ fault, our Elixir apps have just never had the same kind of problems. Maybe we just learned how to architecture things better, but I think Elixir helps to push you in the right direction, and makes it harder to make mistakes. A lot of languages are great, and I’m not saying Elixir is the best language for every circumstance, but for us, Elixir was an excellent choice. I don’t doubt that it’s been a factor in getting Podium where we’re at today. If you need more Elixir in your life, check us out, we’re always hiring.