Elixir in Action author Saša Jurić

Towards Maintainable Elixir: The Core and the Interface

In my previous article, I presented the development process used at Very Big Things. Now it’s time to turn our attention to the code. As developers, we spend a lot of our time inside the source code, so it could be argued that it is an important part of our working environment. To do our job efficiently we need our environment to be well organized. In development, this is the role of code design.

The general idea is to split the code into parts that are mostly independent of each other. If we’re able to achieve that, we’ll get the code that is easier to work with. Understanding how some functionality works won’t require juggling with thousands of lines of code inside our mind. Instead, we can focus on a reasonably small part of the codebase. This will allow us to focus on the task at hand, such as modifying some functionality, adding a new feature, debugging, or optimizing. In addition, good code organization will assist the team dynamics, making it easier to work on the code written by someone else, switch to a completely different project, or onboard new team members.

In this article, I’ll explore our approach to code organization and detail the high-level design, leaving further refinements for future posts.

Core vs interface

A typical VBT backend is powered by Phoenix, so at the highest level, we use Phoenix contexts to organize the code. In our projects, we treat contexts as the core of the system, and web as the interface. The core is responsible for all tasks that must be done regardless of how the system is accessed by the external clients, while the interface layer contains all the logic specific to the way the clients access the system, such as REST, GraphQL, or WebSocket. In other words, the core implements the desired behaviour of the system, while the interface layer exposes the system to external clients.

Such design helps us reach our stated goals. When a developer wants to understand something about the system behaviour, they can go straight to the core, ignoring all the specifics of routing, decoding/encoding, HTTP error codes, etc. On the other hand, changing something in the interface, or even supporting a whole new protocol (e.g. adding GraphQL next to the existing REST API) can be done with no knowledge of the core internals.

At the abstract level, the distinction between the context (core) and the web (interface) is aligned to the approach explained in the official Phoenix docs. However, at the implementation level, we make some departures from the “blessed way.” This is best clarified with the code, so let’s see some examples:

A core function

Suppose we want to implement a user registration functionality. The core function could have the following signature:

Type specifications are our primary documentation tool, and they also help us validate the design. Combined with good naming, a well-focused spec can be very informative. In this example, the spec clearly states the required inputs of the register operation and the possible outcomes. For VBT projects, we aim for precise typing of exported functions, which means not only that type specs must be included, but also that types must be specific. Broad types, such as map and any, are treated as code smells. Ecto.Changeset.t is usually only returned as a part of an error tuple.

This is our main departure from the official Phoenix docs which propose a slightly different approach along the lines of:

The problem with the “official” approach is that the interface concerns leak into the core, blurring the border between the two layers. This defeats the main goal of the design: clear separation between the layers.

The signature of the core function is not particularly informative. We’re not much smarter after reading it, so we need to dive deeper into the implementation. Likewise, reading the controller code won’t tell us the entire story about the interface. Tasks such as checking that required parameters are present, discarding unsupported parameters, casting to correct types, are all performed inside the core. But these tasks are specific to the interface. Why? Because they are needed only for weak protocols, such as REST, while other protocols, such as GraphQL, can ensure compliance of the input data according to the given schema.

Such reasoning is our main tool for assigning responsibilities to a layer. If some problem is protocol-specific, then it is an interface concern. On the other hand, if the code needs to run in all the cases, regardless of the protocol used, it is most likely a core concern.

Let’s see the implementation of the core function:

The most important thing to note here is validations. The core function will check if the password is strong enough and if the e-mail is unique. These are the examples of the rules we enforce at the core layer because these rules have to hold no matter how the system is used.

Note that we don’t separate the domain flow from the database operations. This approach is somewhat impure, but it’s good enough for VBT projects since the domain logic is typically not complex enough to justify the extra effort and bureaucracy of maintaining a pure domain logic. More generally, the core is more than just a business domain. This layer also deals with concerns such as persistence, and communications with supporting 3rd party services. If needed, the core can (and usually will) be further divided into different parts, but this is a subject for another article.

Implementing an interface function

Let’s turn our focus to the interface layer next, choosing a standard Phoenix controller as an example. A controller action will accept two parameters: the Plug conn, and the action params. The latter is a free-form map with the following properties:

• Keys are strings
• Some keys might be missing
• Unknown keys might be present
• Values might be untyped (e.g. if query string format foo=bar&baz=2 is used)

The role of the interface is to convert such input into a well-typed structure required by the core. To make the distinction from the core-level validation, this process is called input normalization. The interface is responsible for normalizing the possibly weakly-typed input, while the business-level checks are left to the core.

For example, the core register function requires an e-mail and a password. There’s no way you can invoke it without those pieces of information, so the client code must somehow produce those values, typically by fetching them from the user’s input, reporting an error if they are not present. However, not every email or password are necessarily valid. E-mail uniqueness and password strength are business-level constraints, and so they are enforced inside the core.

In controllers we usually normalize the input data with schemaless Ecto changesets:

I’ve seen some developers arguing against this approach, claiming that controllers should be “thin” and immediately delegate to context. But then, the context becomes too fat, and all benefits of code separation are lost. Our approach requires a bit more LOC and bureaucracy, but in return, we get better-focused layers, which is the goal of the design.

The usage of Ecto in the controller may seem a bit contradictory to the context pattern. Ecto mainly deals with database concerns, which is clearly a responsibility of the core. However, Ecto.Changeset is a database agnostic abstraction focused on normalizing and validating the data, which is precisely the challenge we’re facing here. As a rule, the usage of Ecto.Changeset is allowed in the interface, while other Ecto functions are not. This is enforced with the boundary tool. Having a clear division of responsibilities between the interface and the core makes it easy to explain such decisions.

Working with schemaless changesets can be slightly noisy, so we’ve made a small wrapper that allows us to normalize the input in a more declarative fashion, somewhat similar to the functionality natively supported by GraphQL schemas and absinthe:

In terms of LOC, this isn’t shorter at all, though for larger input schemas there will be some minor saving. However, the real benefit of this approach is that the input schema is consolidated and presented more clearly.

Activation mail

Let’s see a slightly more nuanced example. We’ll extend the registration functionality by including an activation e-mail. The e-mail has to be sent regardless of how the client is interacting with the system. The choice is therefore clear: sending an e-mail is a core concern, so somewhere in the core we’ll add a function called send_activation_email.

The question is who should invoke this function? One option is to have the interface code invoke it after register:

But the problem is that we’d need to repeat this in every interface. Therefore, a better option is to send an e-mail from inside the register function. This approach leaves no room for mistakes, and properly models the absolute truth that no matter how the user registers, an activation e-mail will be sent to them.

The e-mail content doesn’t depend on the interface, so composing the body is a core concern. Composing textual content can be done elegantly with Phoenix views and template. Consequently, we end up using some Phoenix modules inside the core. Again, the clear separation of responsibilities makes such decisions easy, dismissing subjective views that composing the message is a “UI concern” or that Phoenix modules shouldn’t be used in the context. Views and templates are helpers for building text, so their usefulness is not limited to just web servers.

The e-mail body will contain an activation link, which is composed of the site URL, some fixed paths (e.g. /activate), and a user-specific activation token. The first two pieces of information are the interface concerns, while the last one is the core concern. Combining these three parts into the path is a concern of the interface, but we need to do this during the core operation. So how can we reconcile this? Our approach is to introduce the core contract for providing URLs:

The core function takes the implementation module as the parameter:

The interface layer implements the contract:

Finally, the client can invoke the core function:

It’s worth mentioning that in our projects we actually send the e-mail asynchronously. Instead of immediately communicating with the mailer service, we create a persistent job, powered by the wonderful Oban library. Since creating a job is also a db operation, we need to run everything inside a transaction:

In this version send_activation_email will insert the job record in the database, while the e-mail will be sent asynchronously after the transaction is committed. With this approach, we also get the retry logic for free, since Oban will retry the failed job after some (configurable) backoff time.

The function Repo.transact is our small wrapper around Repo.transaction. This function commits the transaction if the lambda returns {:ok, result}, rolling it back if the lambda returns {:error, reason}. In both cases, the function returns the result of the lambda. We chose this approach over Ecto.Multi, because we’ve experimentally established that multi adds a lot of noise with no real benefits for our needs.

Summary

This post revolved around our understanding of web vs contexts. Our goal is to organize the code in a way that allows people to focus on a single, reasonably small piece of the system. To make this happen, we try to establish a clear separation between different parts of the codebase.

At the highest level we use the standard Phoenix context pattern, slightly tweaking it by treating the context as the core, and web as the interface layer. This design requires a bit more lines of code, but in return, we get better clarity and focus. The code becomes easier to work with, which helps the team keep the long-term sustainable pace, and smooths out possible disruptive effects of team membership changes.

Separating the core from the interface usually won’t be enough. Further organization inside each layer will be required (more on this in future articles). But in my personal view, this separation is the basic minimum that should be done in any long-term real-life project. Without it, the code is going to be much harder to work with, as I’ve personally witnessed many times. On the other hand, this pattern is less useful, possibly even counterproductive, in small one-off projects such as prototypes and experiments.

It’s critical to constantly keep the ultimate goal in mind because then the choices become much clearer. We’re not striving for some academic code purity, nor do we aspire to win architectural awards. We’re in the business of making software and delivering business value to our clients. To do this efficiently, we need a well-organized working environment, and the design presented here helps us achieve that.