Distributed Elixir on… Heroku?

Heroku has a well-deserved reputation for abstracting away many of the complexities of managing and scaling web applications in a robust and flexible way. But to enjoy those benefits, you have to make certain accomodations in the design of your application architecture — stateless nodes, no attached storage, no private networking… you know, most of the things that you want to take advantage of in an Elixir application. Technically, you can get the networking, if you want to pay A LOT for it, in their private spaces offering.

But even though Heroku may not be the perfect environment for running stateful Elixir apps, there’s still plenty to like about it, especially if you’d rather work on anything other than devops. So, if you’re running an Elixir app in Heroku and you still kind of wish you could get a remote shell into a running node, this post is for you.

Before I begin, there’s probably better ways of going about some of this, as I’ll point out. But I’ve been tinkering at piercing the routing mesh to get into a named dyno for awhile, so it was fun to finally get this working in any form. I think it can be improved, and I’m hopeful that we’ll see easier ways to get nodes talking to each other. This procedure will get you a remote shell into an Erlang or Elixir node, but its not going to magically give you that Heroku cluster we’re all hoping for.

So the thing that makes this all possible is that Heroku recently put into beta a new feature called Exec which lets you connect to a named dyno. You can run bash, copy files, and (crucially for our purposes) forward a port. There’s a bit of a drawback to the last one though, mainly because you can only forward a single port, and distributed Elixir requires at least two — one for epmd and another for the node itself. (A SOCKS proxy is also provided, but I wasn’t able to figure out how to make that work with my setup.)

So first step, install ps:exec following the instructions on Heroku’s docs.

$ heroku plugins:install heroku-cli-exec
$ heroku ps:exec
$ git commit -m "Heroku Exec" --allow-empty
$ git push heroku master

Once you’ve got that first bit done, you can test it with heroku ps:exec -d web.1 bash to get a shell into your web.1 dyno.

Next we’re going to need to get around that pesky limitation of only having a single port to forward. This article from Erlang Solutions was really the key getting this to work. You really should read all of it, but if you’re in a hurry, drop this file into your application directory and compile it with elixir epmdless.ex, creating several .beam files in your root. (This is less than elegant, but there’s no part of this that isn’t kind of hackish, so no reason to stop now.)

[Side note: The reason we’re going this route is because I couldn’t figure out how to make SOCKS work for this — if you manage to get that working, drop me a line.]

So now we’ll need to change our Procfile to start up a node with the network enabled. First, set an environment variable that contains the cookie for your nodes to share:

$ heroku config:set BEAM_COOKIE=<cookie value>

Next up, the Procfile changes: web: elixir --erl "-proto_dist Elixir.Epmdless -start_epmd false -epmd_module Elixir.Epmdless_epmd_client -setcookie $BEAM_COOKIE" --name $(echo $DYNO | sed -e 's/\.//')@127.0.0.1 -S mix phoenix.server --no-halt

Let’s unpack that a little:

• We’re starting an Elixir node;
• replacing the default epmd with a client and distribution that calculates a port from the node name;
• setting the cookie from the environment variable;
• setting the node name to the dyno name (which comes from the DYNO environment variable, but we have to remove the period (the sed bit);
• and finally running the phoenix server task.

Commit that and push it up, and you should have a node you can connect to.

The last step is to connect a local node to that one. First, start your port forwarding: heroku ps:forward 4371

(If you followed along with the Erlang Solutions post, remember that each node will add its node number to the base port of 4370.)

Next, start a local node with the same options, but call it node0 instead of using the DYNO setting:

$ iex --erl "-proto_dist Elixir.Epmdless -start_epmd false \ 
-epmd_module Elixir.Epmdless_epmd_client \ 
-setcookie <cookie value>" --name node0@127.0.0.1
Erlang/OTP 19 [erts-8.2] [source-fbd2db2] [64-bit] [smp:8:8] [async-threads:10] [kernel-poll:false]

Interactive Elixir (1.4.1) - press Ctrl+C to exit (type h() ENTER for help)
iex(node0@127.0.0.1)1>

And test to make sure you can get to the remote node:

iex(node0@127.0.0.1)2> Node.ping :"web1@127.0.0.1"
:pong
iex(node0@127.0.0.1)3>

I am finding that if I issue the command after waiting too long, the tunnel dies when I try to connect and i get a :pang response instead of :pong. Just restart the tunnel and immediately issue the command again in your IEx shell. You can also tail your logs with heroku logs -t which may give you some additional information if you need to debug what’s going on.

Now, the moment of truth: Change shells using Ctrl-G iex(node0@127.0.0.1)4> <ctrl-G>
User switch command -->
r 'web1@127.0.0.1' 'Elixir.IEx' -->
c Interactive Elixir (1.4.1) - press Ctrl+C to exit (type h() ENTER for help)
iex(web1@127.0.0.1)1> <ctrl-G>
User switch command -->
j 1 {erlang,apply,[#Fun<Elixir.IEx.CLI.1.81644508>,[]]}
2* {'web1@127.0.0.1','Elixir.IEx',start,[]}

Notice the node name? We’re now on the dyno node (web1, and we can further prove it to ourselves by listing the jobs!

Play around a bit, but when you’re done, don’t forget to exit the shell correctly or you’ll kill your dyno: Go back into user switch mode, and kill your remote shell by k <n> where <n> is the job number of your remote shell.

Ok, what’s next?

There’s definitely some improvements to be made to this. I’d like to get the SOCKS bit figured out, but I’d also like to look at automating nodes doing this for themselves and discovering which other nodes are out there and automatically connecting themselves. This will almost certainly be a subpar solution especially as regards things like latency, as the traffic will have to travel outside the Heroku routing mesh to do that. But, its a start, and I’m hopeful Heroku will eventually open up some version of private networking between nodes to facilitate stateful app platforms like Elixir.

In the meantime, I’m looking forward to being able to get into and debug production nodes when I need to. If you’ve got other ideas, please leave them in the comments!