Our first Elixir project
The good, the bad, the future
Last month, we released our first Elixir project. I wish I could tell you that everything went smoothly, but it was more of an emotional roller-coaster where I went from crazy happy to totally stressed, finally reaching a desired zen mood.
The project
The project was quite simple: creating a marketing site for a company that sells products that could potentially save lives. Having a client with such a good mandate encouraged me to do my best, and beyond. By coding their marketing site, I had the feeling that I would also be helping save lives if they succeeded. This was coding for a reason, in a time when developers are starting to question the ethics of their jobs.
We used Elixir on this project for many reasons, namely:
- open telecom platform (OTP) as a selling point;
- experimenting to see if Elixir would be a good fit for our company;
- embracing the future through the use of a new coding language and paradigm.
OTP as a selling point
Elixir relies on Erlang and its OTP. In short, OTP is a layer on top of the system that lets you play with processes safely. You get parallelism, safety, and fun. For this project, this was ideal as it offered the opportunity to have a caching system that’s solid enough to power the site. We’ll get to it later, but this also created a nightmarish situation.
We used Contentful to power the site with data. Contentful is a content management service that lets you model the data the way you want, in a way that fits best with the requirements. The backend application is thus a bridge between the user’s browser and Contentful’s API.
Technically, every request that targets Contentful goes through an OTP process that returns the last result immediately from the cache, via an in-memory data store called Erlang Term Storage (ETS) and in parallel an HTTP request is made to Contentful to update the cache with the latest available content.
I consider this a sub-optimal compromise to do the caching because there are more efficient ways to go about it. For example, relying solely on the cache and using Contentful’s web hooks to update the cache is an option.
Using the same technology to do the caching is ideal. Parallelism will also help you save you on your infrastructure budget. Elixir is a great solution to be able to take advantage of this key feature for any high-traffic online retailer.
Phoenix, the good surprise
Like many agencies, we build a lot of applications in Ruby/Rails. Personally, I find Elixir to be a welcome alternative. Here’s why:
Adaptability
Rails and Ruby are ruled by conventions. These conventions sound great, and in some ways actually are. A model is placed in a model directory, a controller in the controller directory, etc. When I started with Rails, I had a background in Java (Android, GWT and J2EE applications). So Rails came as a relief because everything was much simpler. However, it seemed that Rails had forgotten all the lessons learned by the Java community in terms of code organisation. Say adios to simple Ruby objects, or to your repositories. And who will miss monstrosities like Hibernate or Maven? Not me, for one.
On the other hand, Phoenix took the good lessons from Rails, as well as its best conventions to help start a project and share common ground amongst developers. At first, this scared me, because what would be the point of switching then?! Fortunately, Elixir has this lovely particularity in that file content is independent of its filename and thus independent from its path. This allows the developer to have the freedom to shape the code in terms of the business domain:
- If it’s a simple app, stick with Phoenix template.
- If it’s a more complicated app, reshape it.
I’d like to give a special shout out to Gary Rennie for his talk at the recent Elixir London conference about how structuring code away from Phoenix-established conventions can make it more sustainable on larger projects. If you like hexagonal architectures, this is for you.
Smooth transition
As my coworkers would say, functional programming is just a “bag of functions” namespaced correctly, in other words, a double-edged sword. It’s easy to write these functions, but beware of the fallacy of writing them in the wrong place.
Overall, our development phase and the transition from Rails went quite smoothly. It was relatively easy to help the various front-end developers working on the project. The more everyone can be autonomous, the better. In this case, using Phoenix with Rails conventions helped a lot.
Deployments
Last April, I went to the Empex Conference with the sole motivation of learning how the heck to deploy an Elixir project. When it came to finally deploy our project, I had this one sentence from Pete Gamache looping in my head: “I’ve been in the trenches for a little bit and.. I’ve seen good men die”. You can count me among those men!
So far, I’ve been able to try three unique deployment methods:
Heroku
This is fast, easy to setup and reliable. It’s limited if you aim to create a cluster of nodes but perfect for simple projects. It would be my favourite solution if we used mostly Heroku at Dynamo.
Edeliver with Distillery
This is a bit harder to setup. There are some constraints such as the no `Mix` reference in the code. Writing an upstart script to restart the application whenever the server restarts is preferable. It features a convenient command line. And it’s easier to get data from the running application or crashes.
Cloud66 with Docker
Lastly, Cloud66 is pretty easy to setup, although harder to get live info. It’s reliable in terms of deployments, with acceptable downtime. This is our preferred choice because it’s the solution we use most often to deploy non-Rails apps. Remember to remove verbosity in deployments, or they will fail (it took me a long time to figure this out).
Some people also use AWS to deploy their apps. I don’t really have any recommendations for this. I suggest picking a deployment method that fits best with your company infrastructure and DevOps skills.
The fall
So in our case, sometime in the hours post-deployment, our application crashed. We saw a “time out” in our log and thought Docker’s capability settings were to blame. This turned out to be an incorrect assumption, which cost us a week. During the time that it took us to figure out a solution, CRON was our saviour (don’t laugh!).
It turns out that we ran into this issue: our HTTP requests had a timeout of eight seconds by default, but the cache layer that we implemented had a default timeout of five seconds. The accumulation of unconsumed requests ultimately caused hackney to abandon all the requests, leading to a system-wide server error.
This sounds obvious to us now , but since we hadn’t run into this situation during the development of the project, it took us by surprise. Read: maximum stress.
At the end of the day
Changing technologies and ecosystems is not necessarily a quiet path. There’s always a learning curve to hike, but it’s rewarding. DHH’s point stands true: if you want a quiet night, stick with Rails. These days, Rails behaviours are highly predictable, which is definitely a valid reason to stick with it. If you want to go past Ruby’s limitations, however, I encourage you to explore other avenues and get out your comfort zone. As I said before, I wish it were easier, but it’s definitely worth it.
Have you experimented with Elixir yet? How did it go?
