A Complete Guide to Deploying Elixir & Phoenix Applications on Kubernetes — Part 1: Setting up Distillery

At Polyscribe, we use Elixir and Phoenix for our real-time collaboration and GraphQL API backends and Kubernetes for our deployment infrastructure. In this series, I’ll walk through the setup we used from start to finish to create a system that supports the following:

• Automatic clustering for Elixir and Phoenix channels
• Auto-scaling to respond to spikes in demand
• Service discovery for microservices, including those in other frameworks like Node.js
• Maintaining the exact same environment between staging and production and easily deploying from staging to production
• Relatively easy to setup and manage

Other posts in this series — Part 2: Docker and Minikube, Part 3: Deploying to Kubernetes, Part 4: Secret Management, Part 5: Clustering Elixir and Phoenix Channels

In this part, we’ll create a new Elixir/Phoenix application and set up Distillery. Distillery is an Elixir package that turns your application into a self-contained release, allowing you to easily deploy your application without installing a bunch of dependencies on the target machine. It also comes with a binary that lets you easily start/stop/daemonize your application, perform hot-code upgrades, and connect a REPL to your running application.

To start, create a new Phoenix application using mix phoenix.new myapp and follow the instructions to set up your database and start your application. Alternatively, feel free to follow along with your own Elixir application and modify as necessary.

To use Distillery, start by installing the Hex package. In your mix.exs, add {:distillery, “~> 1.0”} to your deps and run mix deps.get to fetch the package.

Next, run mix release.init to generate the rel/config.exs file. This file configures the releases for various environments.

Let’s try building a release to make sure Distillery is configured properly. Run mix release. If it’s successful, you should see Release successfully built! along with the commands to run the release in interactive (ie. using iex), foreground and daemonized modes. If Distillery informs you that you’re missing applications, add them to your applications list in mix.exs. This is necessary because Distillery will include only those dependencies (as specified by the applications list) that are necessary to run your Elixir application into its release.

Next, let’s modify our config/prod.exs file to use environment variables for configuration instead of hardcoding the configuration directly into our files. This has two main benefits:

1. It allows us to use the exact same release for staging and production which keeps the differences between the two environments to a minimum
2. We can avoid storing any credentials in our repository and later we’ll use Kubernetes’ secret management to securely pass them to the application

An important caveat is that when Distillery builds our release, the config/* files are evaluated at compile time, not run time. This means we can’t just use System.get_env/1 to get the environment variable value since it’ll pull the value out of the environment on our development machine. Instead, we’re going to use template strings to which will automatically be replaced by environment variables at run-time as long as the environment variable REPLACE_OS_VARS=true is set.

Modify config/prod.exs to look like this:

use Mix.Config

config :myapp, Myapp.Endpoint,
  http: [port: "${PORT}"],
  url: [host: "${HOST}", port: "${PORT}"],
  cache_static_manifest: "priv/static/manifest.json",
  secret_key_base: "${SECRET_KEY_BASE}",
  server: true,
  root: “.”

# Do not print debug messages in production
config :logger, level: :info

# Configure your database
config :myapp, Myapp.Repo,
  adapter: Ecto.Adapters.Postgres,
  hostname: "${DB_HOSTNAME}",
  username: "${DB_USERNAME}",
  password: "${DB_PASSWORD}",
  database: "${DB_NAME}",
  pool_size: 20

The changes we made from the default config/prod.exs were:

1. We no longer import prod.secret.exs since we’ll be using environment variables to configure secrets instead a secret file not stored in version control
2. We copied the configuration formerly in prod.secret.exs into prod.exs
3. We modified the Endpoint and Ecto configurations to use “${SECRET_KEY_BASE}”, “${HOST}”, “${DB_NAME}”, “${DB_HOSTNAME}” etc, which are template strings that will be replaced at run time with the environment variables of the same name. (Note that we didn’t set the pool_size this way. This is because of a limitation of this templating method that only allows us to set string values while pool_size requires an integer. If you need to workaround this limitation to set different pool sizes in staging and production, see the post [TODO: post about setting pool size])
4. We added server: true to our Endpoint to automatically start serving when the application starts so that in production we don’t require execution of the phoenix.server task
5. We added root: “.” to our Endpoint to configure the application root for serving static files. Note that this and cache_static_manifest can be removed if Phoenix isn’t serving any static files (eg. you’re building an SPA hosted separately and Phoenix only provides an API). See the Distillery documentation for more information on hosting static files.

Next we can try building a production release using the environment variables for configuration. Run MIX_ENV=prod mix release --env=prod. If it’s successful, you should see once again see Distillery’s Release successfully built! message.

We should now be able to set the environment variables required in the config/prod.exs and run the release. We need to make sure to also set the environment variable REPLACE_OS_VARS=true so Distillery replaces our template strings. Once those are set, we can run the release with ./_build/prod/rel/myapp/bin/myapp foreground.

$ MIX_ENV=prod mix phoenix.digest # Only necessary if we have static files
$ REPLACE_OS_VARS=true PORT=4000 HOST=example.com SECRET_KEY_BASE=highlysecretkey DB_USERNAME=postgres DB_PASSWORD=postgres DB_NAME=myapp_dev DB_HOSTNAME=localhost ./_build/prod/rel/myapp/bin/myapp foreground

That’s it! We now have Distillery set up building a production release configured by environment variables. You can use Ctrl-C to gracefully terminate the Phoenix application.

In the next parts of this series we’ll go through creating a Docker image for our release and deploying it to Kubernetes along with secrets and automatic clustering.