Erlang Behaviors

…and how to behave around them (2/2)

Continuing with what we saw in the previous post, we’ll talk about what happens when you decide to define your own behavior.

Motivation

Before we start with the tips, let’s take a second to consider why and how we decide to define a new behavior in the first place. In my experience, the reason that leads to the best results in this scenario is the one that lead OTP Team to define their behaviors: to abstract common functionality. This happens when you already have structures, patterns, functions, etc. shared among multiple modules of your system and then you decide to separate the common bits into a generic module or library and leave the specific ones in the callback modules. In the upcoming paragraphs I’ll refer to this practice for creating behaviors as deduction.

The opposite scenario is also possible: You may already know that you will have to implement multiple versions of the same thing so you decide to start from day 0 with the definition of a behavior for that. This is what happens when you’re creating a new library/framework that you want your users to implement. As you will see below in the list of tips, if this is the path you’re following you should be very careful and either very thorough or very flexible because what you’re doing is, in part, futurology. You’re guessing/deciding what the use cases and requirements from your users will look like even before having actual implementations for them. I’ll refer to this practice as induction.

These two approaches are not exclusive, in fact they’re usually combined. After deduction, you’ll probably want to expose your new behavior to some other users or start building more callback modules yourself. That generally means you’ll need to do some induction and consider what those future uses may look like, how will they deviate from what you found so far. Conversely, after induction you’ll will want at least to show that your new behavior works or how it works. You’ll probably have to write one or two callback modules and, just by using them, you’ll end up improving you behavior definition applying a bit of deduction at this stage.

Now, to the tips!

Tips for Defining Behaviors

Disclaimer / Introduction

Same as last time: the following tips are all heuristics, not rules. There are exceptions and situations in which is much more convenient to bend or break these guidelines to better accommodate to your needs.

1. Definition & Usage Together

As a general rule: behaviors should be defined (i.e. -callback attributes should be written) in the generic module.

There is an analogous limitation to the fact that all callbacks should be implemented in one module: all callbacks should be defined in one module as well, and there is no mixer library to help you here (I never needed one, anyway). In principle then, the generic module is the only one who knows about those callbacks.

Also remember that for a callback to be used, at some point a dynamic call should be evaluated (something like Module:the_callback(Param)). Dynamic calls are (at the time of writing this article) impossible to analyze by tools like dialyzer or xref. That’s why elvis has a rule to warn you if you use dynamic calls on modules that don’t define behaviors.

If you really only need to define a behavior to be sure that all your types provide similar interfaces, don’t be afraid to add functions like the one below to your generic modules:

-module(figure).
-type t() :: #{module := module(), _ => _}.
-callback area(t()) -> float().
-export([area/1]).
-spec area(t()) -> float().
area(Figure = #{module := Module}) ->
  Module:area(Figure).

As you can see, the function area/1 is just a middleman. Its only goal is to allow other modules to evaluate figure:area(…) with any figure created in a callback module that implements the figure behavior. The alternative would be to force callers to figure out the callback module and evaluate the dynamic call themselves.

2. Callback Specs

Put lots of effort on having proper -callback specs. This will be easier if you’re doing deduction because you will clearly know the types that you need to accept/provide. But it’s even more important if you’re doing induction since specially in that case, they should act as documentation and also allow dialyzer to correctly type-check callback modules.

The following callback provides no documentation and almost no type specification for dialyzer:

-callback init(term()) -> {ok, term()} | {stop, term()}.

You can add documentation like this:

-callback init(Args :: term()) ->
            {ok, State :: term()} | {stop, Reason :: term()}.

And that’s how gen_server defines its callbacks. I personally prefer to do a bit more:

-type init_arg() :: term().
-type callback_state() :: term().
-type reason() :: term().
-callback init(init_arg()) ->
            {ok, callback_state()} | {error, reason()}.

That way if we later need to use the same type for other callbacks (and that’s generally the case with callback_state/0), we can precisely indicate that it’s in fact the very same type and not just another term(). This practice also has the added benefit that, in the callback modules, the specs will look like this:

-spec init(gen_mod:init_arg()) ->
        {ok, gen_mod:callback_state()} | {error, gen_mod:reason()}.

That way, dialyzer can check that the types exist and, if we ever change the definition of any of them (e.g. we may want to use maps for init/1 argument) we only need to change it in one location (on gen_mod). We don’t need to change every callback module spec.

3. Testing

Going back to what I said on the previous article: In general, the generic module only works if it has a counterpart (i.e. the callback module), the generic module is not useful on its own. So, to test your generic module, you’ll need to create at least one implementation of the behavior and test it in black box mode. This is clearly easier when you’re deducting since you should already have tests for the modules that you’re refactoring.

It’s more complex for induction scenarios. In those, you might want to generate a fake callback module (or even more than one) and use them for your tests. You can also provide a default implementation of your behavior and test this one. This is particularly useful if you’re writing a library. Check sumo_db, for instance: This library provides multiple behaviors to extend the different layers of its architecture (i.e. repository, store, backend, etc.) but it also comes with an implementation of all of them (the ones based on mnesia). That way, other developers may have a clearer reference on how they’re supposed to implement their adapters. This is information that goes beyond what the callback specs can provide.

Not only that, but sumo_db also provides test helpers that developers can use to ensure compliance and test coverage of their modules without the need to copy&paste the existing tests that are used for the mnesia versions.

4. Optional Callbacks

Erlang/OTP 18.0 introduced optional callbacks. Those who follow my blog should know that I am a huge fan of them :) — but I’m here to tell you that they should be used with extreme caution.

If you mark a callback as optional you need to make sure that your generic module works well with a callback module that does not export said callback. That means that if your default or fake module (the one you’re using for your tests) does implement the callback, then you need to use another fake callback module without the function and test your generic module with it, too.

Just in case that you need it, remember that you can use erlang:function_exported/3 to find if the callback is implemented, but only after the callback module is loaded. If you’re using releases (and we all are!) all application modules will be automatically loaded on startup. That’s great, but if you’re writing a library, you can’t be 100% sure that your users will use releases to run their Erlang systems, can you? ;)

5. Using Existing Behaviors

Finally, remember that you don’t need to start from scratch every time. The same module can be a callback module for a behavior and define another. If an existing behavior takes you half the way there, you can just build your new behavior on top of it, like OTP does ;)

Conclusion

Behaviors are a deceptively simple yet powerful tool that is one of the most important additions that OTP provides over plain old Erlang. Understanding how to use them properly is a must if you want to build an Erlang system.

But you can do more than that, you can build your own behavior and you can share it with the community, just like the OTP Team did. That will give you an even more power. Just use it with care, because you know…