Deploying Phoenix Applications with Exrm
Introduction
Doing all of this work for web development in Phoenix has been wonderful so far, but eventually we actually need to push our code into production, one way or another. We could, of course, take the simple way to get it into production: take the current codebase, scp it to our server (or pull it down via git onto the production box), set up our database, and start running our application with:
PORT=80 MIX_ENV=prod mix phoenix.server
But that’s not exactly the best way to get this all working, and it also throws out a lot of the neat features we get running on top of BEAM, such as hot code reloading for zero-downtime deployments! Given that, how can we get our code into production in a way that’s not prone to making anyone cry?
Simple: We’ll use Exrm! We’ll start off with our blog engine that we’ve been working on in the “Writing a Blog Engine with Elixir and Phoenix” series. If you have not been following along but want an application to follow this guide with, you can grab the codebase from https://github.com/Diamond/pxblog (use the part_3 branch for best results if you’re following along directly, as that is the point at which this guide was written/followed along).
One important thing to note, however: you must use the same architecture for building your release that the release is getting deployed to. If your development machine is OS X and you’re deploying to a Linux server, you need a Linux machine to build your exrm release or it isn’t going to work, or you can just build on the same server you’re going to be running everything on.
The deployment process that we’re going to follow is that we’re going to build the release on the production server directly. We’re going to rely on the fact that we’ve been checking this in to github to make our deployments relatively smooth.
Setting up: adding the exrm dependencies
We’ll start by adding exrm to our list of dependencies for our application in mix.exs, under the deps private function:
And then we’ll run mix do deps.get, compile and everything should just work! To make sure everything is working, we’ll make sure that we have some new mix tasks for our current application:
mix help | grep release
mix release # Build a release for the current mix application.
mix release.clean # Clean up any release-related files.
mix release.plugins # View information about active release pluginsSetting up: preparing our application for deployment
We need to modify one piece of our config/prod.exs file so that Elixir knows that this app, when it’s deployed and running, should function as a server, so we’ll add another key to our config :pxblog section, setting server to true:
Now, you’re going to need to SSH into your server and clone your repo/checkout the right branch. Once we’ve cloned the Diamond/pxblog project and checked out the add_exrm_for_releases branch, we’ll cd into that directory and begin. You’ll need to quickly run mix do deps.get, compile to make sure everything is ready. Next, we need to make sure that our static assets have already been compiled and are ready to go for our server, so we need to run:
MIX_ENV=prod mix phoenix.digestExcept doing so right now will fail for us because we don’t have a config/prod.secret.exs file! This file should not get checked in to any source control (and won’t, by default (for Mix), with git), so if you’re pulling down this repo, you’ll be missing it too. The prod.secret.exs file should contain our secret key base as well the database connection details. I’ll post an example here:
If you need to generate a new secret key base, you can use the following mix command:
mix phoenix.gen.secretThe output from that is a key base you can just copy and paste into your prod.secret.exs file. Change your database details to whatever is valid for your deployment box and save the file. Now you can try again to run:
brunch build --production
MIX_ENV=prod mix phoenix.digest(If you get an error message about “The input path priv/static does not exist”, just quickly make that directory (mkdir priv/static))
If all goes well, you should get a message about “Check your digested files at priv/static”, and we’re finally ready to actually build our release!
Including Conform
If you’re working with a separate system administration department, and you told them they’d need to edit some Elixir files to configure the server, odds are they may not be very happy with you. Thankfully, there is another dependency we can pull in to make this a bit more palatable for everyone involved, and it’s called Conform. Conform generates a configuration file for your release that uses simple dot notation and makes it very simple to modify configuration details as necessary.
We’ll start by modifying our mix.exs file to include conform as a dependency.
And then run mix do deps.get, compile to make sure everything is back up to date. Now that we have conform added as a dependency, we can see what it has added to mix. Run the following command to get more information:
mix help | grep conform
mix conform.configure # Create a .conf file from schema and project config.
mix conform.effective # Print the effective configuration for the current project
mix conform.new # Create a new .schema.exs file for configuring your app with conform.Conform gives us three tasks by default: conform.new (which generates the schema file needed to build out the .conf file), conform.configure (which generates the .conf file from our schema), and conform.effective (which prints out whatever the current application configuration is set to).
One important thing to note is that, at least in my experience thus far, I had to run these commands with MIX_ENV=prod or else they would fail. Since you’re generating this for a production release, that makes sense anyways!
Given that, we’ll set up our schema first:
MIX_ENV=prod mix conform.newAnd we should get a success message telling us the schema for our project has been placed in config/[app name].schema.exs. Next, we generate our conform configuration file:
MIX_ENV=prod mix conform.configureThis time we should get a success message telling us that the .conf file for our app has been placed in config/[app name].conf. Finally, we’ll quickly output the result to test that everything went as expected.
MIX_ENV=prod mix conform.effectiveOne very important thing to note, however: the generated .conf file IS NOT in your .gitignore by default. You either must add it or never check in/push from your build box! Alternatively, you can leave it checked in but only use that for staging boxes and override the conf file on your actual public production box(es).
One thing that is still missing, however, is that we don’t get a port specified in our [app].conf file. Look for the following lines:
# Provide documentation for [app].Elixir.[App].Endpoint.http.port here.
# [app].Elixir.[App].Endpoint.http.port =Notice that it is both blank AND commented out. Right now, Conform does not appear to work with environment variables in the prod.exs file (the bit about {:system, “PORT”} gets ignored completely, as does using port: System.get_env(“PORT”). I haven’t figured out why this is yet, but thankfully there’s a very simple fix: uncomment that line and choose which port you want to use for that application in production. If you do not do this your application unfortunately will not start, and will give you an error message about port being blank.
Generating our release
Our last step is to run the following commands:
MIX_ENV=prod mix releaseIf all goes well, you should get a message similar to the below:
==> The release for pxblog-0.0.1 is ready!
==> You can boot a console running your release with `$ rel/pxblog/bin/pxblog console`
We’ll do just that. One difference is that we’ll need to specify the port this should run on, so the full command should look like:
PORT=4001 rel/pxblog/bin/pxblog consoleWhich should drop us into an IEx console! The exrm release command generates a binary that includes a ton of commands (just run rel/pxblog/bin/pxblog with no arguments to see them all). The ones we’ll use the most frequently are:
pxblog start - Start up the server running on the specified port
pxblog stop - Shut down the server
pxblog restart - Restart the server
pxblog ping - Make sure the server is still running
pxblog remote_console - Open up an IEx console into the currently running server processIf we go into rel/pxblog/releases, we should see a bunch of files that were created as the end result of our call to mix release. If you do not see a file labelled pxblog-0.0.1.tar.gz, then just grab 0.0.1/pxblog.tar.gz and copy it into rel/pxblog/releases as pxblog-0.0.1.tar.gz.
Running migrations
Running migrations is a lot like what we’d do during development, so that’s pretty simple. This is assuming, of course, that you’ve already set up the database that you intend to use for your application so that all that’s remaining are the migrations.
MIX_ENV=prod mix ecto.migrateWe should see all of our migrations run.
Starting up our service
I created an /apps directory to store all of the applications that I’m working with. I created a pxblog-exrm directory to store my release files for this application. I then copy the created tarfile into this /apps/pxblog-exrm directory:
cp ~/releases/pxblog/rel/pxblog/releases/0.0.1/pxblog.tar.gz /apps/pxblog-exrmUntar the file we just copied and finally, we’ll start our application. We’ll pick a port (I’ll pick 4040 for no particular reason):
PORT=4040 /apps/pxblog-exrm/bin/pxblog startReleasing an upgrade
Let’s say that after we make this change and run it we notice that the title in our web/templates/layout/app.html.eex file doesn’t actually match what we want (it still says “Hello Phoenix”). Let’s fix this quickly and hot deploy an upgrade to our application without tearing anything down (yes, seriously)! Open up app.html.eex and change the title, and then open up mix.exs and change the version number from 0.0.1 to 0.0.2. Now, it’s time to generate a new app:
MIX_ENV=prod mix phoenix.digest && MIX_ENV=prod mix releaseThen grab the 0.0.2 release tar file that was created and copy that over to our release directory again:
cp rel/pxblog/releases/0.0.2/pxblog.tar.gz /apps/pxblog-exrm/
tar zxvf /apps/pxblog-exrm/pxblog.tar.gzAnd now we’ll run the command to run a live upgrade:
/apps/pxblog-exrm/bin/pxblog upgrade 0.0.2
Using /apps/pxblog-exrm/releases/0.0.2/pxblog.sh
Release 0.0.2 not found, attempting to unpack releases/0.0.2/pxblog.tar.gz
Unpacked successfully: “0.0.2”
Generating vm.args/sys.config for upgrade…
sys.config ready!
vm.args ready!
Release 0.0.2 is already unpacked, now installing.
Installed Release: 0.0.2
Made release permanent: “0.0.2”And that’s it! Refresh your browser pointing at your server and you should see our title changes! That was a pretty sweet zero-downtime deploy!
Rolling back an upgrade
What if we didn’t like the change (or it introduced more problems than it fixed)? Well, exrm includes a “downgrade” command as well to…well, downgrade an application! Just like the upgrade call, the only argument it takes is which version to downgrade to.
/apps/pxblog-exrm/bin/pxblog downgrade 0.0.1
Using /apps/pxblog-exrm/releases/0.0.2/pxblog.sh
Release 0.0.1 is marked old, switching to it.
Generating vm.args/sys.config for upgrade…
sys.config ready!
vm.args ready!
Release 0.0.1 is marked old, switching to it.
Installed Release: 0.0.1
Made release permanent: “0.0.1”Opening up a console to the service
Finally, we may need to do some work that requires a live console window. We can open up an IEx shell that is connected into our application with the following command:
/apps/pxblog-exrm/bin/pxblog remote_consoleIssues I ran into
Ubuntu has problems compiling apps with exrm by default
If you’re running Ubuntu 14.04 on your build/production machine, you’ll want to install erlang-eunit and erlang-base-hipe so that you don’t get compiling failures or “Missing beam file elf_format” error messages while building your exrm releases.
Blank assets (no CSS/js being loaded on the site)
Fixed by making sure Brunch was installed and running the following commands:
rm priv/static/*
brunch build --production
MIX_ENV=prod mix phoenix.digest
MIX_ENV=prod mix releaseNo specific version file being generated in [app]/rel/[app]/releases
For whatever reason, I am not getting the versioned tar file in pxblog/rel/pxblog/releases. Instead, I can see a tar file getting created under pxblog/rel/pxblog/0.0.1/pxblog.tar.gz. This isn’t too big of an issue as I just copied that file to where I’m running my apps, but it definitely does not match up with any of the documentation I read on exrm.
Deployment with migrations are still super confusing
Right now, we have to copy up the source files so we can run our migrations on the server. There’s a lot of discussion on how to handle the whole migration deal and I haven’t seen anything conclusive yet. Still waiting on this one! The good thing about this particular deployment method is that if you’re like me and developing on OS X but deploying to a Linux-based server, you’ll either have to do this or have a build machine/VM/docker image to use in the middle.
This is still pretty manual
Right now, this is a little manual for a deployment process and I haven’t really sat down to improve my own deployment process. I could theoretically use something like mina or capistrano to script the deployment process, but since this was the first time through there were a lot of gotchas I wanted to make sure were properly handled. One solution to this that I still need to explore is edeliver.
One possible deployment strategy
If I were using this for something more mission critical, I would likely not scp the source to my production server and build/deploy. Instead, I’d likely set up an intermediate build server that would also have access to the database server. I’d SCP the files there, build the server, etc, and also run the migrations there too. The deployment process would instead look like this:
[Development Machine pushes to Github] -> [Script pulls down code onto Build Machine from Github] -> [Build Machine creates Exrm release] -> [Build Machine runs deployments] -> [Build machine SCPs release to production environment] -> [Production environment runs/upgrades]
That allows me to tune the roles that the production database accounts have access to, allows me to not have to install all of the development dependencies on my production box(es), and keep a pretty sane deployment path.
Need a place to deploy to?
Charles Watson wrote up a great piece on getting an Elixir-capable environment up and running using DigitalOcean, so if you’re looking for a place to deploy your new application (that isn’t Heroku), take a look at his post! https://medium.com/@SirCharlesWatson/setting-up-digital-ocean-for-elixir-and-phoenix-a17dd44aaf21
Getting an error message about “unreachable_package”?
I ran into an issue on a build server about an unreachable_package, where the unreachable package was actually the full Phoenix application that I was running! It started off with me getting errors about “ERROR: Failed to build release. Please fix any errors and try again.”. To debug what was going on, I started off by using the verbose flag for mix release.
MIX_ENV=prod mix release --verbosity=verboseThis helped me piece together that the release process was dying somewhere in particular. The error message that I ended up getting was:
===> Provider successfully run: app_discover
===> Running provider resolve_release
===> Solving Release pxblog-0.2.1
===> Provider (resolve_release) failed with: {error,
{rlx_prv_release,
{failed_solve,
{unreachable_package,
pxblog}}}}This was a pretty frustrating error message and I spun my wheels for awhile trying to figure out how to correct this one particular error message. The trick that worked for me (and may not work for you, so YMMV warnings apply here) was running this command:
MIX_ENV=prod mix do deps.get, compile, releaseThat appeared to be the magic trick to get this message:
==> The release for pxblog-0.2.1 is ready!
==> You can boot a console running your release with `$ rel/pxblog/bin/pxblog console`So, if you’re ever getting any wonky error messages, use the verbosity flag on mix release to figure out what is going on!
