Intro to Distributed Elixir
How Elixir and the BEAM can make distributed systems easier
Some time ago, your overly excited co-worker came around talking about Elixir and the BEAM, a platform that made hard things easy. You signed up for all the things. You’re using pattern matching to make your code more readable. You’re using pipelines and tokens to make composable functions a snap. You’re using Supervisor trees to make your app more fault-tolerant and self-healing. You’re even doing hot code upgrades! You’re living the dream.
Something is nagging you though. This Distributed Erlang thing they kept mentioning… It sounds cool, but it felt… unnecessary. In this post, we’ll look how using distributed Elixir can supercharge your applications for FREE.
What is distribution?
Said simply, distribution is running an application across more than one server. Why would you do this? Why deal with the added complexity? Splitting your application makes it easier to scale a specific part of your offering, makes the code easier to reason about since you have smaller code bases, and lets you handle situations of total server obliteration. Failure in a distributed system looks like a hard drive failure, or an entire region being unavailable.
Distribution in Elixir
Elixir and the BEAM are built for scenarios where failure is normal. Supervisors and their restart strategies are first class citizens. Distribution is also a first class citizen and can be used without additional plugins, and very little configuration. In addition, by connecting additional Elixir nodes, you get a ton of feature upgrades for free.
Phoenix Presence
In this great talk by Sophie Debenedetto, she covers how to set up Presence. If you are running multiple connected Phoenix servers though, you get automatic syncing of Presence data for free! No external dependencies, no nothing.
Mnesia
OTP gives you a relational database out of the box called Mnesia. If you have multiple connected applications, your Mnesia database will be automatically distributed and replicated across your cluster! AMAZING.
Plain old Elixir Apps
Connecting Elixir applications lets you send messages to pids across applications as if they were locally created. This means you can start writing an app to run on one server, then add distribution without having to change much code! SO GREAT.
How does it work?
Named nodes can communicate with each other with a shared secret. The simple example is to open 2 terminals like so:
# terminal 1
iex --name terminal1@127.0.0.1 --cookie 'super_secret'# terminal 2
iex --name terminal2@127.0.0.1 --cookie 'super_secret'# terminal 1’s iex Session
iex(terminal1@127.0.0.1)1> Node.connect(:"terminal2@127.0.0.1")trueiex(terminal1@127.0.0.1)2> Node.list[:"terminal2@127.0.0.1"]iex(terminal1@127.0.0.1)3> Node.spawn(:"terminal2@127.0.0.1", fn ->...(terminal1@127.0.0.1)3> IO.inspect Node.self()...(terminal1@127.0.0.1)3> end):"terminal2@127.0.0.1"
So what’s going on here? We’re starting two sessions with that shared cookie of super_secret. We’re then connecting the nodes with each other by using Node.connect/1. We need to use that funky syntax since the input has to be an atom. After they’re connected, you can run functions on any connected node. Here we’re calling IO.inspect Node.self() on my other terminal. It could very well be another machine!
One thing to note is that once a node is connected, any new nodes it learns about will be propagated to the cluster. Say you have terminal 1, 2 and 3. If 1 connects to 2, and 2 connects to 3, 1 will know about 3 and vice-versa. Just another one of those things the BEAM makes easy.
Libraries for clustering
Setting up clustering is pretty easy, but we can do better with libcluster. libcluster gives you different strategies for discovering and clustering your applications. It’s very useful if you’re deploying to Kubernetes.
Another library to look at is partisan. partisan is there if you need better performance out of your distributed system. It’s an alternative to the built in clustering support built into the BEAM.
What can go wrong?
Network setup
The BEAM runs a process (epmd) that runs on port 4369 that is the first stop for your node when it’s trying to connect to a sibling. If this port is closed, you’re not getting distribution going. Make sure that port is accessible from your other servers. Some platforms make this a bit harder, but ask your local networking guru for help.
Split brain
This is a tough one. Networks are inherently unreliable. You may not notice it but network communication is riddled with dropped packets, timeouts, and malformed responses. Now that we have these clustered apps, what happens when they lose sight of each other? Worse yet, what happens when multiple servers take in input they thing is “true”? I put a bid in on server 1, but you put a bid in on server 2? In a perfect world, our distributed setup would handle this, but what if server 1 and 2 are disconnected? Who should server 3 believe?!
While not an easy problem, there are algorithms like Raft that try to solve these “consensus” problems. There’s even an implementation for Elixir built by the RabbitMQ team called ra.
Distribute all the things
Elixir and the BEAM make hard things easy. Next time you’re building an app, ask yourself, “How can distribution make this better?”
Happy Clustering!
Thanks for reading! Want to work on a mission-driven team that loves Elixir and making hard things easy? We’re hiring!
To learn more about Flatiron School, visit the website, follow us on Facebook and Twitter, and visit us at upcoming events near you.
Flatiron School is a proud member of the WeWork family. Check out our sister technology blogs WeWork Technology and Making Meetup.
