Runtime configuration, migrations and deployment for Elixir applications

Shortly after moving from PHP to Elixir I’ve faced a common issue, the way how do we deploy applications is totally different from the one I’m used to.

After running few production I want to share my experience and best practices with newcomers to reduce entry barrier for deploying our lovely Elixir code. This article is a little bit opinionated but I will try to list alternative approaches in case you want to explore them. And this is not step-by-step guide, you will need to figure out missing pieces.

Releasing your code

I saw few guides that recommend you to simply copy your code to the server and run something like mix phx.server, but it’s really bad to do that.

To run your application in production you need to create an OTP release. Currently, there are few ways that allow archive this goal but most common and community-tested tool is Distillery.

Distillery docs are comprehensive and include the view of it’s developer on how you need to deal with runtime configuration. I recommend you to read it and pick whatever works best for you.

What is an OTP release?

OTP release is a set of applications that should run together and their configuration. It may include Erlang Run-Time System (ERTS) in case you want to deploy to the server that does not have pre-installed ERTS or it’s version mismatch from the one that you used to build the release.

The first thing you need to keep in mind is that releases are platform-dependent, you can’t build a release on you macOS laptop and use it on Ubuntu server. I will describe in the following sections possible ways to address this issue.

The second one is that lot’s of your code would be resolved at compile time, so you need to pick a configuration strategy which works for you.

Adding Distillery to the Elixir project

It’s very similar to adding any other Elixir dependency:

1. Add {:distillery, "~> 1.5", runtime: false} to you mix.exs;
2. Fetch dependencies by running mix deps.get;
3. Add release configuration to the rel/config.exs, for most of my app it like this (simplified):

There are few things that a require explanation:

• There is only one release environment. I’ve never faced a need to build an OTP release with separate settings for a development environment, so they are always built in the same way.
• I do not include ERTS since we deploy code via Docker containers or to Heroku, that have ERTS pre-installed. This reduces the tarball size.
• Erlang distribution cookie should be strong random, anyone who knows that cookie and has a network access to the server would be able to run any commands on it. Do not use the same value for microservices that should not communicate via Erlang distribution protocol to ensure that would not connect to each other with defaults.
• You can generate rel/config.ex with defaults by running mix release.init.

List of all available options is available at Distillery docs.

After completing this steps you can run MIX_ENV=prod mix release and use the release tarball which can be found at _build/prod/rel/myapp/releases/0.1.0/my_app.tag.gz, where prod, myapp and 0.1.0 should be replaced with you Mix environment, app name and version respectively.

$ ls -l _build/prod/rel/myapp/releases/0.1.0
drwxr-xr-x   4 andrew  staff       136 Aug 18 17:28 commands
-rwxrwxrwx   1 andrew  staff     16472 Aug 18 17:28 myapp.bat
-rw-r--r--   1 andrew  staff    148764 Aug 18 17:28 myapp.boot
-rw-r--r--   1 andrew  staff      2051 Aug 18 17:28 myapp.rel
-rw-r--r--   1 andrew  staff    205330 Aug 18 17:28 myapp.script
-rwxrwxrwx   1 andrew  staff      3714 Aug 18 17:28 myapp.sh
-rw-r--r--   1 andrew  staff  18496269 Aug 18 17:28 myapp.tar.gz
drwxr-xr-x  10 andrew  staff       340 Aug 18 17:28 hooks
drwxr-xr-x   8 andrew  staff       272 Aug 18 17:28 libexec
-rw-r--r--   1 andrew  staff     19921 Aug 18 17:28 start_clean.boot
-rw-r--r--   1 andrew  staff      4358 Aug 18 17:28 sys.config
-rw-r--r--   1 andrew  staff       549 Aug 18 17:28 vm.args

Configuration

Since the by default configuration is resolved at compilation time, you need to take care of it or your DevOps would have a bad time trying to run your app.

Why? It’s hard to have exactly the same production configuration environment on the build server, configuration tends to change and it’s bad when you need to recompile and deploy everything to change a database password. That’s only a few of many other the reason because of which more and more people are trying to follow 12-factor app guidelines.

Another issue is that you would have all your secrets compiled into release artifact and everybody who has access either to it or to the build environment would know actual production secrets.

Phoenix suggests to use config/prod.secret.exs for storing production secrets, I don’t like this approach since it makes configuration even more complicated and does not problems listed above. In all the projects we removed include for it in favor of fully configured config/prod.exs for all deployment environments. But I know people that avoid using environment variables entirely. Instead, they generate prod.secrets.exs via Ansible that is used to deploy an application to the standalone VM’s.

I want to start with few examples of things that would not work unless they are build on with actual production configuration:

1. @my_attr System.get_env("FOO"). Attributes are resolved at compile time, so the actual value would be nil if you don’t have FOO environment variable on the build server or its value in case it’s present there. Do not use attributes for configurable values.
2. config :myapp_api, :key, System.get_env("FOO"). The same here, System.get_env("FOO") would be evaluated at compile-time during config.exs conversion to sys.config. By the way, I recommend checking sys.config when something is not working after release due to configuration issues.
3. Your app is no longer a Mix project, so you won’t be able to do any Mix.* calls. You can include :mix as application dependency, but don’t expect it to work properly.
4. Project structure would be changed to match OTP styles, without taking care about /priv dir you won’t be able to run migrations.

Luckily there are plenty of options how to deal with these, from simple to the sophisticated ones.

Application.put_env/3 in start/2 callback

This is probably the simples way which really works if you have very few configurable values. Whenever Elixir application is started, Application.start/2 callback is triggered, we can leverage it and set configuration value resolved from system environment before supervisors are started:

Updating the environment before application supervisor is started

Notice: Don’t try to set those values for other applications (eg. to set it for :sample_web from :sample_lib). There is a chance that invalid value would be resolved because you won’t know in which order they are started.

REPLACE_OS_VARS env

This is the simplest way to start using environment variables for configuration, simply set REPLACE_OS_VARS=true in your target environment and replace all actual values in config/prod.exs with ${VAR_NAME}. Here is a little catch — you can’t use it for non-string values, so it is not possible, for example, to set pool_size for Ecto.

Here is an example:

Usually, you would fall to this method when you don’t have control over the code that is using configuration, eg. when using some third party library that doesn’t have init callback, support for {:system, _} tuples or passing all required options on function calls. There is a good article by Michał Muskała for library developers on this topic.

Init/2 callbacks

init callbacks for Ecto and Phoenix allows you to define a function that can use custom logic to resolve configuration at application start. For me, this is the most convenient and useful way to configure a library.

Example for Ecto:

You can notice that we use DATABASE_URL here. Passing a single environment variable is much easier and it’s automatically set in some deployment environments, eg. on Heroku.

:system tuples

For some apps, we use Confex library that allows resolving {:system, _} tuples in init/2 callbacks. It has a nice property—you can set a default value when the environment variable is not set.

When all your configuration can be covered with :system tuples you can have a single config.exs file without the need to worry about adding separate config macros in all per-env config files. Sometimes you would simply forget to set it and pay for it with debugging time.

Example for Phoenix with Confex:

Confex allows you to write a custom configuration adapter, which makes it possible to use third-party tools for configuration management, eg. Hashicorp Vault.

Using Module callbacks

You can set an @on_load callback in any Elixir module, it will be invoked whenever the module is loaded. Although this is not a common way to configure your application, it’s useful when refactoring code that reads configuration in the attribute, eg.:

Can be effectively rewritten to:

Don’t worry about Application.get_env/2 for being a bottleneck, it uses an ETS table with read concurrency.

Notice: Modules compiled with HiPE can not use this module callback.

Distillery Config Providers

In version 2.0 (released on Aug 2018) Distillery introduces a new way to configure your application when it starts — Config Providers. They can be used to populate Application environment during boot process. I highly recommend read the docs on this topic since it’s one of the ways where Elixir community may go as a new defacto standard how to configure your applications in production.

Accessing priv directory

You probably have some migrations and other stuff inside /priv directory, to access it use :code.priv_dir(app) in your Elixir code and OTP machinery would make sure you won’t loose files inside of it.

Running migrations on production

After making your release configurable, you would also need a way to run migrations. Even though there are few options how you can run them (eg. connecting to the node and issuing an RPC command), adding a release tasks seems most useful for me.

First, you need to add a ReleaseTasks module to your app, here is an example that we use:

You can call migrate/0 and seed/0 functions via Distillery commands /path_to_release/bin/myapp command “Elixir.MyAppAPI.ReleaseTasks” migrate. Notice that module name is prefixed with Elixir. because after releasing we need to use Erlang-style module names and all Elixir code lives within this namespace.

Since this the command itself looks over-complicated, let’s add two shell scripts to the rel/commands/ as a shorthand to run them:

Add these scripts to rel/commands and run chmod +x rel/commands/* in you shell

After that, we can add a Distillery commands to make everything simpler:

Now you can run migrations via /path_to_release/bin/myapp migrate command.

Deploying Elixir apps

Deploying to Heroku

After finishing all preparation in sections above it’s really easy to deploy your app to the Heroku.

1. Create an app with Elixir buildpack:

$ heroku apps:create myappapi-staging --buildpack https://github.com/HashNuke/heroku-buildpack-elixir.git
Creating ⬢ myappapi-staging... done
Setting buildpack to https://github.com/HashNuke/heroku-buildpack-elixir.git... done
https://myappapi-staging.herokuapp.com/ | https://git.heroku.com/myappapi-staging.git

Notice that this app would be our staging env, we will create production a little bit later.

2. Add a Procfile and elixir_buildpack.config to the application source:

Procfile is used to declare process types that are required by your app. web is a special type that will receive HTTP traffic by listening on PORT which automatically is set by Heroku. You don’t have control over which port it listens to and can’t use more than one public port per app.

The release is a release phase that is used to run migrations when an app is built, environment configuration is changed or application is promoted in a release pipeline.

The application is started in foreground mode (/app/bin/contractbook foreground) to keep the Dyno running and to redirect logs to the STDOUT. Similarly to Docker containers, everything restarts when the main process exits or enters background mode.

Also, notice that I am setting post-compile hook in elixir_buildpack.config that creates a release, removes all files that we don’t need in production to keep the released artifact small.

Post compile hook should be placed in rel/hooks/post_compile.sh:

3. Add GitHub integration to pull source code and optionally enable automatic deployments. You can connect master (main) branch straight to the staging app.

3. Add environment variables that your app is using.

4. Optionally enable Heroku Postgres.

Now you can make sure everything works and create a production app by forking staging by running heroku fork — from myappapi-staging — to myappapi or manually with steps above. After doing that update environment to use production secrets.

Add all apps to the Heroku pipeline, they should look like this:

You can also enable Review Apps, Heroku will automatically create a temporary application on each GitHub pull request, which is very useful to run tests versus changed code base. To do so add an app.json to your app source:

Heroku provides an app.json generator when you click on Enable Review Apps.

Now each time code is changed your application is deployed automatically to the staging environment. Probably you want to do that after passing CI tests, don’t forget to set this flag in auto-deployment configs in Heroku UI.

When code in staging is tested you can deliver it to production by clicking “Promote to production” in Heroku UI, via CLI interface or Slack ChatOps bot. Production would use exactly the same release artifact (slug) which guarantees that you don’t have build errors on the production release.

If you want more examples of apps that can be one-click deployed to Heroku — check out Man template rendering engine source code.

Deploying to Kubernetes

Kubernetes is a container orchestration tool used by many productions, including GitHub. From deployment side of view it is similar to the Heroku, but instead of Heroku slug, we need to build a Docker container.

I won’t describe Kubernetes setup, there are lots of great guides for it and I expect that you familiar with it. You can even go the hard way. Most of the production systems I’ve build are running in Google Container Engine (a managed Kubernetes cluster) because we didn’t want to have operational costs to manage the cluster itself.

To build a Docker container add a Dockerfile to your project. Here is an example from Annon API Gateway:

As you can see, I worry a lot about container size so we are using our own Apline Linux docker container. This example leverages multi-stage builds introduced in Docker 17.06 CE.

I don’t want to make this article even bigger, but you can find many samples in Annon infrastructure repo, including yaml files for Kubernetes deployment.

YAMLs can also be improved by rewriting them to Helm Charts.

Deploying to a standalone VM

I’ve never gone this path since operational costs for us never outweighed the benefits of this approach. (See comparison table below.)

The main benefit of it is that you can upgrade your code without downtime. You would definitely value it if your app has long-living connections (eg. WebSockets for IoT devices) that are expensive to restart.

Checkout edeliver package. It has all information that you need to get started.

Deployment targets comparison

If you still not decided where do you want to deploy your apps, here is a small comparison table that I’ve made for myself:

Want to know more?

There is a really good book that I recommend you to read by Francesco Cesarini — “Designing for Scalability with Erlang/OTP”. It has a section that covers OTP releases in-depth and provides much more useful details that would help you to archive proficiency in Elixir.

Second recommended read is “Learn You Some Erlang for great good” (free online version) by Fred Hebert.

And ask you questions in comments. I am going to write more articles on topics that are requested by the community.

Thanks

I want to thank Andrew Summers for proofreading this article and OvermindDL1 for pointing out the right way to get path to priv.