Elixir — Process Registries

Process Registries are an important concept in Elixir / Erlang. In this post, we will go over what process registries are, why we need process registries in the first place, and how to implement a process registry. Implementing a process registry will give us an opportunity to dive into the Erlang docs. I am always excited when this happens — in my experience, learning from primary source materials often leads to a fuller understanding of the problem at hand. Let’s get started!

What is a Process Registry?

A process registry is a mechanism that maps keys to process IDs. A process registry aids in process discovery. Remember that, in order for Process A to communicate with Process B, it must, ultimately, know Process B’s pid. A process registry is a mechanism by which Process A can discover Process B’s pid from a string, a tuple, or any other valid Elixir term.

(Note: We have used the word key above, but in the context of process discovery, we would often use the term alias. Don’t let that confuse you with how Process.register/2 is used — while Process.register/2 requires the key / alias to be an atom, part of the feature-set of a process registry is that any term can be used as the key / alias. I use key here to make sure we understand that, conceptually, a Process Registry is a map that takes a key and gives you back a value; that value happens to be a process id.)

Why Do We Need Process Registries?

The need for a process registry in Elixir / Erlang systems occurs as a result of two design decisions.

Firstly, at the VM level, we have already mentioned that communication between processes occurs via message passing. In order to send a message to a process, the sender must, ultimately, know the pid of the process it wants to communicate with.

Secondly, at the application level, we use supervisors to achieve fault-tolerance. Supervisors are responsible for overseeing a set of processes. Should any of these processes crash, the supervisor will start a new process in its place. That new process has a different process ID than the process it replaced.

This leads to a problem. Let’s say that Process A needs to periodically communicate with Process B. A simple solution would be for Process A to hold on to the pid of Process B, and use that to communicate with Process B. Now, imagine that Process B has crashed — someone tried to divide 1 by 0 again! Luckily for us, Process B was being supervised, and the supervisor started a new process in the old process’s place. Great, except that Process A still has a reference to the old Process B’s pid. We could, of course, hand-roll a solution such that Process A receives an updated pid whenever Process B dies and is restarted. But what happens when we have 100 processes with a pid referencing Process B? Instead of keeping track of and updating every process that needs to communicate with Process B, let’s make that update to only one process, and have all other processes ask that source of truth for the pid of the process they need to communicate with, whenever they need it. That source of truth is the Process Registry.

So, to recap, a Process Registry simply maps valid Elixir terms to process IDs. A Process Registry’s role is to keep a consistent, up-to-date mapping of terms to process ids to aid in process discovery. This is necessary because supervised process can be restarted, thereby changing the process IDs of different parts of the system.

A Pitstop in Erlang Docs

Normally, after providing an understanding on what a concept is and why it is necessary, we would jump into implementation. However, when I was first learning about process registries, the various materials I found on the topic all required the exact same API, down to the function names and return values. I found this bewildering at first, but the reason why the API was so consistent across sources is that a process registry is every bit as much a behavior as GenServer, even though there is no Elixir ProcessRegistry module or Erlang :process_registry. Don’t let the fact that there is no abstraction that implements a process registry behavior fool you — we are dealing with a behavior / pattern here. Therefore, we will dig into the Erlang docs to figure out what Erlang requires the API of a process registry to be.

Finding the Right Docs

Given that there is no abstraction on the process registry behavior, our challenge is to get to the right set of docs so that we can learn how to implement the behavior. How do we do this? Well, I did have one important piece of information — eventually, we’d be using the process registry as part of the arguments to :gen_server.start_link/3, so I started to look at the docs for that function, and came upon this piece of information:

If ServerName={via,Module,ViaName}, the gen_server will register with the registry represented by Module. The Module callback should export the functions register_name/2, unregister_name/1, whereis_name/1 and send/2, which should behave like the corresponding functions in global. Thus,{via,global,GlobalName} is a valid reference.

Now we’re starting to get somewhere. According to the docs above, we’re going to have to implement four functions, and the behavior of those functions should be the same as global. global is actually a reference to an Erlang module. Not knowing that at the time, I simply Googled for what I thought would be the most unique search term — i.e., “global unregister_name” — and was brought to the right set of docs. At this page, we finally have our API — four functions, with their name, arity, and their potential return values.

Why the Diversion Into the Erlang Docs?

The purpose of this digression was two-fold. First, we need to understand why we have to implement our process registry in a certain way. A Process Registry is, again, a behavior — a pattern that we must abide by if we want to use our registry with other OTP patterns, such as GenServer.start_link/3. This pattern determines what functions we implement, how we name those functions, what argument those functions are invoked with, and what the return values of those functions are.

Secondly, being able to navigate and read documentation is the developer’s equivalent of “learning how to fish.” Not only will this help you answer your own questions, but it will allow you to ask better questions when the docs are not enough. By putting in the effort of doing your own research, as much as your current skills allow, others will be more willing to help. As Chris Rock said:

I’d always end up broken down on the highway. When I stood there trying to flag someone down, nobody stopped. But when I pushed my own car, other drivers would get out and push with me. If you want help, help yourself — people like to see that. — Chris Rock

Reading the docs is how you “push [your] own car.” People will be more helpful when they see you gave an initial stab at figuring things out yourself.

Now, back to our regularly scheduled programming.

How to Implement a Process Registry

Earlier in this post, I mentioned that a process registry is a mechanism for process discovery. There are a multitude of ways to make this abstract concept concrete, but in this post, we will be implementing our process registry as a GenServer.

The fact that a Process Registry is a pattern makes things very easy for us in terms of implementation, but let’s go over the high-level requirements of the Process Registry:

1. Our process registry must be able to register a process — i.e., associate a given pid with a given key, regardless of the key’s complexity.
2. Our process registry must be able to deregister a process.
3. If we ask our process registry for a key, it must be able to return the correct pid.
4. Finally, our process registry must be able to, automatically, deregister a process when that process dies, so that its state is consistent.

Creating the GenServer (Step 0)

First, let’s create the skeleton of our application — i.e., a GenServer that holds some state. Given that the purpose of our process registry is to map keys to values — where the values happen to be pids — it makes sense to use a Map as the internal data structure.

Adding a Term / PID Mapping (Step 1)

Now that we have our Process Registry GenServer, let’s add functionality that “saves” a term / pid into our process’s state. In other words, let’s implement register_name/2.

Let’s walk through the code.

First, on Lines 9–11 we create a register_name/2 function that “mimics” :global.register_name/2, for reasons we’ve already discussed. If you are familiar with the GenServer concept, there is nothing too surprising here.

Next, on Lines 18–27, we implement a callback function that will check if the key we are trying to insert already exists. If it doesn’t, then we add that key / value pair to our registry, and reply back to the client with the atom :yes. If the key already exists, we don’t do anything except reply to the client with the atom :no. We will come back to Line 21 later in the post; you can forget about it for now.

(Note: Notice that the fact that we used a call instead of a cast, and the fact that we return :yes and :no atoms was not left for us to decide, but were a result of following the functionality of :global.register_name/2. This will be recurring throughout the rest of the functions we implement).

Removing a Term / PID Mapping (Step 2)

Our process registry must also implement an unregister_name/1 function that deletes a term / pid pair:

The code needed to remove a term / pid pair is fairly straightforward. On Lines 13–15 we create ProcessRegistry.unregister_name/1 function, which sends a message to our Process Registry server (:registry), and simply deletes the term / PID from its state (Lines 22–24). Note that if the key doesn’t exist, then Map.delete/2 returns an un-altered map. This frees us from having to put in any conditional logic.

Return a PID Given a Term (Step 3)

Now that we can add and delete term / pid, we need to be able to ask our Process Registry for the pid associated with a given term:

The addition to our Process Registry in order to support this feature is similar to unregister_name/1. We’ve added to the Client API ProcessRegistry.whereis_name/1 (Lines 17–19), and we’ve added the callback to handle that message on Line 41–43.

The only real item of note here is that Map.get/2 will return the atom nil if it doesn’t have a pid registered under the given key. However, the :global.whereis_name/1 must return either a pid or the atom :undefined, so we had to pass specify that in our call to Map.get/3.

Maintaining a Consistent State in the Process Registry (Step 4)

Let’s go back to Line 21 from Step 1 / Line 33 in Step 3. As I mentioned before, one of the crucial functionality of a Process Registry is that it serves as the source of truth in terms of what processes are and are not registered with it. This means that, if a process dies, we must remove it from our registry.

In order to do that, we ask the Registry to monitor any processes that are added to it. A monitor is a uni-directional link. In our case, if the process being monitored dies, it will send a message to our registry’s inbox. (This differs from a link in that a link is bi-directional. Here, if our process registry where to die, the monitored process will not be notified).

In order to respond to that message we need 1) the shape of that message and 2) a handle_info function that successfully pattern matches against the shape of the message.

How do we figure out what the shape of the message is? Take a look at this quick asciinema video to see how we can use iex to answer this question, and others like it. The message we will receive will look something like this:

{:DOWN, #Reference<0.0.3.73>, :process, #PID<0.59.0>, :killed}

Therefore, we will create a callback function that pattern matches on that shape:

Let’s talk about the helper functions first. We have defined a deregister/2 function in our module, but notice that the first function will look for a pid as the second argument, thanks to the guard clause. (Remember that the order of multi-clause functions is important).

If we try to deregister a pid, then the function on Line 52–57 is invoked. This function iterates through our term / PID map, and looks for the pairing in which the pid is the same as what we have passed in. If it finds a matching pid, it calls deregister/2, but with the moniker, not the pid. This means the second clause of deregister/2 would be invoked.

Notice that we’ve also changed the callback on Line 26–28 to delegate to our helper function, simply so that we can have the logic in one place, rather than scattered throughout our code.

Now, with the helper functions explained, let’s take a look at Line 45–47. This is simply our handle_info function. When our Process Registry receives this message in its mailbox, we know that that a process that our registry is monitoring has died. So we take action, and remove that term / PID mapping from our state.

It is important to notice that, on Line 49, we’ve also added a “catch-all” handle_info/2 callback. If you write a handle_info/2 callback, you should also implement the clause on Line 49. If you don’t do this, then messages that your server doesn’t know how to handle will accumulate in its inbox. (Happy to answer why in the comments, but let’s not get sidetracked).

Implementing the send/2 Function

We are nearly done here, but there is one critical piece of functionality missing. At the beginning of this post, the docs that we found through our research indicated that our process registry module needed to implement a send/2 function that works just like :global.send/2. Let’s do that now:

First, let’s take a look at Line 2. Kernel is a module that is automatically imported into every other module. Line 2 is overriding that behavior. We want to import everything inside the Kernel module, except for send/2. Our module will define it’s own send/2.

Next, take a look at Lines 23–31. Again, how we need our function to behave is laid out in the :global docs, so a lot of this was not really up to us. The function is fairly self explanatory. The only items worth pointing out is that, even though we didn’t import Kernel.send/2, we can still use it by using the fully-qualified name of the function. We do that on Line 26. Secondly, notice that if we get back :undefined from whereis_name/1, we need to send back a tuple in a very specific format.

Via Tuples

Now that we have a fully-functioning process registry, you can start registering other processes with it with via tuples. Via tuples, as we’ve already seen from the Erlang docs on the :gen_server module, is a tuple in the form of {:via, RegistryModule, process_name}, and they can be used anywhere that you can use the :name option. For example, in GenServer.start_link/3, GenServer.start/3, Supervisor.start_link/3. In our example, instead of passing in the option as name: :atom, we would pass in name: {:via, ProcessRegistry, key}, keeping in mind that the key can be any valid Elixir term.

Once we have registered our processes with our process registry, we can then use that same via tuple as the first argument to GenServer.cast/2, GenServer.call/3. Those functions will then use the registry module to get a pid, and continue on their merry way.

(Note: You can find further information in the GenServer docs. Use your browser’s find functionality and search for the Name Registration section).

Conclusion

And that’s the end of our journey regarding process registries in Elixir. From a practical standpoint, I hope that you take away from this post the basic points of a process registry — the what, when, where, why and how of it. In the long-term, I hope that our dive into the Erlang docs helps you go through a similar research process whenever you have a question. Primary sources are a great way to learn, as it decreases your reliance on tutorials and blog posts that may be out of date, may not be completely correct, may leave out important caveats and information, or may not be a direct answer to your question. (Of course, these assumes that the primary source docs are better, but that’s usually the case, and it is definitely the case with Erlang docs).