Erlang Behaviors
…and how to behave around them (1/2)
I’ll diverge a bit from my usual article format. Instead of exposing a single funny bit or a simple lesson, this one will be a list of tips and heuristics on how to work with Erlang/OTP Behaviors and, in particular, how to work when defining your own behaviors. This will also be presented in 2 articles instead of just one. Fasten your seatbelt…
Background
Let’s start with a bit of history, which is important to understand both how existing behaviors work and how you should work if you want to define your own behaviors.
Behaviors are a key part of OTP, and the most important ones (those in stdlib, like gen_server and its friends) were all created in the same way: People at Ericsson figured out that they were doing the same thing over and over again and then, following a path that is very common in functional programming in general, they abstracted the common code (i.e. the generic behavior) of their modules to a single module so that they could use it later and just write the specific code needed in each particular implementation. That’s how gen_server was born: it condensed the knowledge gained by writing many many similar server processes, each one coded in its own module but all of them doing more or less the same thing.
Notice that using a module with callbacks is not the only way we could’ve developed generic components (anonymous functions, for instance, are an alternative). The mechanism devised by the OTP team has its pros and cons and some of them will become apparent in the tips below.
A few things to remember:
- Those described above where just the first ones. OTP include many others (just grep for -callback on otp source code). And of course, you can now define your own behaviors. Documentation for that is sparse and may not be the most thorough, but it’s certainly there.
- Even when behaviors where originally intended to be used for processes (quoting the docs: “…Behaviours are formalizations of these common patterns. The idea is to divide the code for a process in a generic part (a behaviour module) and a specific part (a callback module).…”), they can be used to implement any other similarly parameterizable scenario you might think of (e.g. sumo_doc).
- callback modules and generic modules are meant to build together a single entity. When you write the required callbacks for
gen_serverin your module, you’re actually building a server that will be composed of bothgen_serverand the module you’re writing.gen_serveron its own doesn’t work and neither does your module. They both need each other.
Now, to the tips!
Using Behaviors
This article will focus on using already existing behaviors. I’ll leave the tips for creating your own behaviors for the next one.
Disclaimer / Introduction
As usual with this kind of lists, the following ones are all heuristics, not rules. There are exceptions and situations in which is much more convenient to bend or break these guidelines to better accommodate to your needs.
1. Encapsulation
As I said above, in general, when you’re using a behavior you are building a thing that’s composed by both the behavior and your callbacks. When you implement gen_server callbacks, you’re building a server, when you implement cowboy_rest callbacks, you’re building a REST API handler, etc. The fact that you implement that using a generic module is an implementation decision and as such is usually best to hide that from your users. You can do that by providing an API on your callback module instead of relying on others using the generic one directly.
Notice how, from outside of this module, nobody needs to know that it’s implemented using gen_server. Any other module that wants to use it should only relay on kvs:start/0 , kvs:store/2 and kvs:retrieve/1. If you eventually need to start using gen_statem instead of gen_server, you can simply change the internals of this module and move forward easily.
2. Black Box Testing
Following the same line of thought as the previous paragraph where the usage of a behavior is seen as an implementation detail, considering that your tests should not test implementation but functionality and keeping in mind that, if you followed the previous rule, you already provided a nice API for your module… you should write your tests against that API.
In other words, even when handle_cast/2 is exported in your module, you shouldn’t write tests that evaluate it directly. On the contrary, it would be much more convenient to write a test where you start your server and then use it’s API to let gen_server invoke your function.
As you can see, nowhere in our test are we directly testing any of the callbacks. We test them indirectly through the module interface.
The reasons behind this guideline are the same ones as the previous one: if you don’t assert anything about how your module is implemented and you focus on how it should behave, you are free to change the implementation and your tests will let you know if you affected its functionality or not.
Caveat: It is, in general, no longer the case now (thanks to optional callbacks), but it used to be quite difficult to follow this guideline if you wanted to ensure 100% test coverage in your code. Certain callbacks may still be almost never evaluated and hard to trigger. In those cases, you do need to end up writing some white box testing. As the 100% coverage advocate that I am, I tend to put all those tests (the ones that are only there to ensure coverage) in a coverage_SUITE or coverage/1 test case.
3. Dialyzer
Since the introduction of -callback attribute, dialyzer is now able to verify that you’re properly implementing the behavior that you want to implement. It’s a great tool that you should use from day 0 on your projects. You can even add it directly to your test suites using katana test.
4. Opaque State
Behaviors (particularly those used for processes) tend to have some sort of state or context passed through functions. That state should be kept hidden from the rest of the world and that’s because it’s an implementation detail strictly tied to the behavior we choose to implement.
Records are great to ensure this, since they’re bound to the module that defines them. Since OTP19 you can achieve a similar effect using well typed maps. In both cases, it’s convenient to define a non-exported type for your state and add proper specs to your exported functions.
Notice how I added types and specs for all functions (with subtypes of what the behavior defines), but I only exported those used in the API functions (i.e. key/0 and value/0). If anybody tries to use any of the other types outside of kvs, dialyzer will properly warn us about that.
5. Mixer
There is one limitation imposed by the generic/callback module implementation: all callbacks should be written in a single module. That means those callback functions can’t be shared between several callback modules. A workaround for that is to use mixer, an open-source library created by Kevin Smith while working at Chef and later on improved by Juan Facorro while working at InakaESI that lets you mix in functions from other modules.
Let’s say we want to implement handling of unexpected messages in the same way for all of our servers. We can write that code in base_server:handle_info/2 then mix that function in all of our servers, et voilà.
Caveat: In the example above, I’m assuming base_server:handle_info/2 ignores its second parameter, so that it doesn’t need to use state/0 type defined in kvs (following Tip #4 in this list). You might need to relax that rule a bit sometimes and share parts of your state types among modules. That’s where maps are really handy. You just need to make sure that the types on the modules that mix in functions from a more generic one use types that are subtypes of the ones used in that other one. For example, the spec of base_server:handle_info/2 in our case can be…
As you can see, kvs:state/0 is a subtype of base_server:state/0 and dialyzer as certainly able to figure that out. You can even use parameterized types there, but that’s a matter for another blog post.
OffTopic Shameless Plug
On October 14th I’ll giving a talk about the stuff I write on this blog (but in Spanish :P) at EmprenDevs. So, if you happen to be around Rosario that day and you want to meet me, register at the website or follow the conference team on twitter or facebook for more information.
