Where is the “application layer”?

When I started promoting my new project ALF (Application Layer Framework), I realized that many people are a bit confused with what the “application layer” is and why we need a framework for that. I can understand this confusion. Many of my friends and colleagues use Ruby and Elixir as the main programming languages and write their code with popular frameworks — Ruby on Rails and Phoenix respectively. These MVC frameworks don’t have a proper abstraction for the “application layer”, so people usually mix domain and application layer logic into a mess of interconnected objects/functions.

In this article, I’m going to illustrate with a simple example what the application layer is and how one can refactor a messy codebase by clearly separating domain layer logic from application layer logic. There is also a suggestion of a missing abstraction in MVC web frameworks.

Layered architecture

I would follow our classic Eric Evans and provide several quotes from his DDD book.

First, why do we need layers?

“In an object-oriented program, UI, database, and other support code often gets written directly into the business objects. Additional business logic is embedded in the behavior of UI widgets and database scripts. This happens because it is the easiest way to make things work, in the short run. When the domain-related code is diffused through such a large amount of other code, it becomes extremely difficult to see and to reason about.”

The motivation is clear — we want to make our programs more simple and clear so that we can easily introduce changes and fix issues.

“There are all sorts of ways a software system might be divided, but through experience and convention, the industry has converged on LAYERED ARCHITECTURES, and specifically a few fairly standard layers. The metaphor of layering is so widely used that it feels intuitive to most developers. … The essential principle is that any element of a layer depends only on other elements in the same layer or on elements of the layers beneath it. … Of course, it is vital to choose layers that isolate the most important cohesive design aspects.”

Eric emphasizes that the separation into layers is “intuitive to most developers”. And I can only agree with that. I would even say that the layered structure is forced by our programming languages since objects/methods or modules/functions are always put into tree-like structures — one function calls other two functions that call the other two and so on.

“Again, experience and convention have led to some convergence. Although there are many variations, most successful architectures use some version of these four conceptual layers: …”

And below is an image from the book:

Let’s go through the layers one by one, but not in the order presented in the image above.

Interface Layer — “Responsible for showing information to the user and interpreting the user’s commands. The external actor might sometimes be another computer system rather than a human user.”

Therefore this layer contains a set of entry-points of a program (functions that initiate a complex business logic underneath and through which data can be entered into the system), and also exit-points (through which data can be presented/retrieved from the application).

Infrastructure Layer — “Provides generic technical capabilities that support the higher layers: message sending for the application, persistence for the domain, drawing widgets for the UI, and so on. The infrastructure layer may also support the pattern of interactions between the four layers through an architectural framework.”

This layer contains all the low-level tools and libraries that a program needs to communicate with the world. And of course, the frameworks we use are also a part of the infrastructure layer for a software engineer.

The application and domain layers are the places we put the core logic of a program. But what’s the difference and why do we need two?

It’s simple with the domain layer since Eric’s book is about that.

“Responsible for representing concepts of the business, information about the business situation, and business rules. State that reflects the business situation is controlled and used here, even though the technical details of storing it are delegated to the infrastructure. This layer is the heart of business software.”

So, basically, all the business logic goes here. What’s left for the application layer then?

“Defines the jobs the software is supposed to do and directs the expressive domain objects to work out problems. … This layer is kept thin. It does not contain business rules or knowledge, but only coordinates tasks and delegates work to collaborations of domain objects in the next layer down.”

It “defines the jobs” and “directs domain objects”, it “only coordinates tasks and delegates work”.

I would use the language metaphor that Eric uses extensively in the DDD book and put the distinction between the domain and application layers in the following way.

The domain layer defines the language of a program — a set of “words” that describes program entities, their relationships, and the actions they can perform, while the application layer is sentences composed of these words. It is a story you tell using the domain language.

That’s easy to say, but in reality, each function, in some sense, “defines the jobs” and “directs domain objects” (data). Also each function, in some sense, is “responsible for representing concepts of the business”. And finally, we use the same programming language for writing code for both layers. In the programming language, we have just data and functions composed into tree-like structures and name-spaced by modules.

An example

For educational purposes, let’s take a look at a piece of logic written in Elixir and try to determine to which layer (domain or application) each part of functionality belongs. I won’t provide the code itself but thought it would be better to provide the “function-call graph” (tree in our case) — a diagram that shows functions and relationships between them. It’s a part of code that fetches “files” from third-party “storage” with “folders” that belong to “customers”, so then “users” of the system can work with the customers’ files.

The flow starts from the `Storage.sync_folders` function that receives an `identifier` of the storage. The function calls `get_credentials` to fetch “credentials” from DB and then calls `sync_folders`.`sync_folders` fetches folders using `FilesAPI` service and then calls `insert_folders_and_files` function (via `handle_response`). It then calls `get_customer`, `build_folders`, and so on and so on. The “DB” label marks the function that interacts with a database, and “TpAPI” shows the interaction with other apps on the cloud (fetching folders, storing files, flagging folders). Green ellipses notate data (entities) that are used on each level: it starts from the storage identifier, then credentials appear, then api_response, folders, users, customer, files.

There is lots of stuff here and the logic is a bit messy. But it’s a good example of some functionality that has been modified by several engineers during a couple of years and is not yet properly refactored. There is nothing particularly wrong with the code from the common “best practices” point of view: there are small functions, limited responsibility, a simple interface of the function, hiding implementation details. The only problem is that in order to understand what is going on, one should traverse the whole tree, and “it’s hard to see the forest for the trees”.

I suggest refactoring it right by identifying functions that can present the mysterious application layer. The “story” metaphor will help us. And the story we are going to tell should be free from technical details and engineering buzz-words. It should be a story that product folks will understand (“ubiquitous language” — the core idea in the DDD book).

First, we have to identify what the actors are in the story. These are our business entities that are represented by corresponding data structures. So our story is about users, folders, and files inside, plus a couple of additional data structures like credentials, customer, api_response. Then, the actions the code does with entities are: fetch data, insert to the database, etc.

One variant of the story is: “Having a storage identifier, first we fetch the credentials record from the database. Then we fetch folders and store them in the database. Then we save files in the cloud and flag folders as processed”. One may wonder about quite low-level technical abstractions here, like “database” and “cloud”. The language you choose depends also on a product manager’s knowledge. If the business folks understand what are the data and where they are stored, it’s ok to have such abstractions in your ubiquitous language. Or you may choose more abstract terms like “repository” to avoid exposure of implementation details.

The story is very simple, there are just 5 actions mentioned and they follow one another. Therefore I would present the application layer as a sequence of 5 functions.

For sure, there might be variations. For example, in the initial code, the “Folders.insert_and_assign_users” function first calls “insert_folders” and then “assign_users”, so it makes sense to add a separate step for it. Also, some additional steps can be mentioned, e.g. “preparing parameters for FoldersAPI”, “fetching customer”, etc. So the number of functions in the application layer can be easily increased to, let’s say, 10. There is no simple rule of where to stop, but I would rather stop with 5 in this example.

And now I just wanna call all these 5 functions into a separate one using the awesome `with` macro:

I put the function into a separate module and used the “use-case” abstraction. Also `get_creds`, `fetch_folders`, and other functions are called in the scope of the responsible context modules: Credentials, FilesAPI, Folders, Files.

I wanna emphasize again that the way you define these core functions depends on the complexity of your domain. For example, the `insert_folders` may be a simple database operation and therefore may be combined with `fetch_folders` into something like `process_folders`. Or on the contrary, `insert_folders` may have a complex logic of how the raw data is processed with tons of additional business rules inside. In that case, you may add an additional step, like `Folders.build_folders_from_response`.

Therefore we’ve not just flattened the “function-call tree”, we’ve also highlighted the “forest”! Now one can easily see what are the top-level functions in the logic and what are the entities these functions operate on. And this is what one can call the “application layer” logic for the case.

And look how the function-call graph looks after the refactoring.

What is missing in our MVC frameworks?

MVC frameworks are very popular among web developers. In Elixir we have Phoenix, in Ruby — Ruby on Rails, other programming languages have their own alternatives. In terms of layered architecture, it seems that “controllers” should be the place when one puts the application layer logic. But it’s not true. In our frameworks, controllers not just call a specific business logic but also are responsible for rendering, and this responsibility actually is forced by the framework itself. Controllers know how to deal with HTTP actions, headers, parameters, and also they render specific data for the “view” layer. Quite a lot of responsibility already.

That is why we were taught from the very beginning that one should keep controllers thin, and what we usually do is just call the top-level function (like “SyncStorage.sync_folders”), thus delegating all the business logic to the “model” layer. And since there is only one (“model”) layer beneath, we usually build quite a weird tree structure without an explicit “application layer”. (The Phoenix framework has contexts and schemas, but both these abstractions belong to the model layer.)

Several years ago, when I worked on a system composed of several Ruby-on-Rails applications, we used a cool small library — “interactor”. It suggests putting parts of the logic into separate “interactor” classes that implement the “call” method, and then execute the interactors one-by-one in the “organizer” abstractions. It looks like:

Each “organizer” has the “context” object — a plain data structure that is accessible from each interactor, thus they can assign and read all the necessary data. An “organizer” is called inside the specific controller action with one line of code and then the result is rendered.

That is how we solved the problem of a missing layer in the Ruby-on-Rails framework.

The same problem exists in the Phoenix framework too. Yes, we have the “context” abstraction, but this abstraction is a “domain” one. As the documentation says: “Contexts are dedicated modules that expose and group related functionality.” It helps us to split the “functional” part of the logic from the “data-related” one. We have schemas with all the changesets, validations, etc, and contexts that operate with these schemas.

Many engineers have a concern about the missing part. There are two good talks about that. The first one is from my former colleague Yevhenii Kurtov who argues that “MVC is not enough”. Another one is from Jenny Shih who used Robert. C. Martin layered approach and introduced the “use-case” abstraction in order to orchestrate domain logic.

I would also support their ideas and would like to see something like “use-cases” or “operations” or “organizers” or whatever place I could put the application layer logic.

And therefore instead of (MVC) I would like to have something like MCCV — model — controller — controller — view, where the first controller layer is a layer for controlling domain logic, while the second one is for the presentation layer (what we have currently). The picture below summarizes the idea.

There are Phoenix abstractions in the middle, on the left — corresponding layers, and on the right — abstractions related to the MCCV pattern that I suggest. Maybe there might be another word for the “controllers” related to use-cases, but I have one argument to support the name. Let’s check how controllers are defined in the original article that suggested the MVC pattern . Keep in mind that it was 1979, long before the web era.

“A controller is the link between a user and the system. It provides the user with input by arranging for relevant views to present themselves in appropriate places on the screen. It provides means for user output by presenting the user with menus or other means of giving commands and data. The controller receives such user output, translates it into the appropriate messages, and passes these messages onto one or more of the views.”

Trygve Reenskaug mentioned several places and views. A controller coordinates (in the general case) several “view” components — sends data to them and triggers the logic. It is very similar to what a “use-case” does — coordinates several underlying components that execute domain logic. This symmetry is the only support of my suggestion to use the “controller” word. Of course, there are many differences, especially if we consider what the views-related controllers do in Phoenix (or Rails) web-frameworks — they fetch parameters for underlying logic and serialize output to send JSON responses to views/FE. So the naming may be questioned.

I hope you enjoyed the reading. Let me know what you think!