Building Software Systems

How to break the software development logjam

A fundamental problem that building software must address is that it is inherently too complex to fit in our heads. Our brains can focus upon only so much detail at any one instant in time — and we are the ones who ultimately have to build the software. To break the logjam, we have to break through the complexity.

Our human brains did not evolve to write computer programs. They are much better suited to other activities. Our otherwise sterling intellectual capabilities were not optimized for the essential complexity of developing software systems — and despite some optimistic claims for artificial intelligence, computers are not yet well-suited to write good software for themselves.

“Many of the classical problems of developing software products derive from this essential complexity and its nonlinear increases with size […] From the complexity comes the difficulty of enumerating, much less understanding, all the possible states of the program”

— Dr. Frederick P. Brooks, Jr., No Silver Bullet

A promising approach to addressing that conundrum is to optimize the division of labor between people and computers. Let humans do what only they can do, while automating the activities that are better performed by computers. But just how do we do that?

Question: How do you eat an elephant?

Answer: One bite at a time.

We must first break the inherent complexity of software systems into bite-sized chunks that our human brains can more easily digest — while not introducing additional complexity as we do so.

Optimizing Focus

Behavioral and cognitive research provides compelling evidence that we are able to focus best on the set of things that is immediately in front of us. Many of our brain’s cognitive functions apparently evolved to ignore the things not in front of us in order to maximize our focus on what was, at the moment, most important for our survival.

In fact, if we have to chase after the pieces of the puzzle that are not directly in front of us — like chasing logic and data definitions across multiple Java classes in multiple packages — we rapidly lose focus and our error rates go up while our comprehension and productivity go down.

We can start by finding component and communications models, and development tools, that help us to isolate and focus on one thing at a time. This can dramatically increase our comprehension and productivity, while reducing the rate at which we make errors — and we all, no matter how accomplished, make errors.

Layered Architecture Limitations

Most of us are familiar with the layered architectural pattern. It is probably the pattern used most often today when building business applications. Many applications that are called monoliths, though perhaps deployed in large monolithic executables, are in fact of the layered pattern — as are most of today’s popular frameworks.

“Each layer of the layered architecture pattern has a specific role and responsibility within the application. For example, a presentation layer would be responsible for handling all user interface and browser communication logic, whereas a business layer would be responsible for executing specific business rules associated with the request. Each layer in the architecture forms an abstraction around the work that needs to be done to satisfy a particular business request.” — Software Architecture Patterns by Mark Richards, O’Reilly.

The layered pattern seeks to break things down into understandable and manageable parts. It is popular because it really does help to do just that.

Unfortunately, the layered pattern’s tradeoffs become less attractive when we try to implement modern agile development practices. Though it rates high in ease of development and testability, it rates low in overall agility, ease of deployment, performance, and scalability.

The layered pattern implements technical partitioning — the layers are broken out by the technologies being used and the technical skills needed to implement them. That partitioning strategy assumes that technology is the hard part of building software.

However, most applications tend to be more about functionality than technology. Especially today, where the technology part frequently comes packaged for us in open-source libraries with well-defined APIs.

Let’s take a look at how we might enjoy the benefits of the layered pattern while overcoming its inherent weaknesses.

The Vertical Slice

We can start by viewing our layered pattern in terms of vertical slices, where we can implement domain partitioning— so we can break things out in terms of application domain knowledge and domain-bounded contexts.

We won’t ignore technology, but we’ll have it serve the application requirements, rather than the other way around.

Vertical Slice of Application Functionality

The vertical slice is a pattern that repeats itself throughout most application features, and cuts a slice through the layers, from external data down to persistent data.

• External data comes in from a UI, an electronic device, an message queue, or another application.
• That data is validated and then computations and transformations are performed as required.
• The results of the transformations are persisted and, or, returned to the UI, device, or external system.
• The UI, electronic device, or external application may retrieve data from persistent storage as needed.

A vertical slice represents a logical, but not necessarily physical layering of a discrete application behavior and the communications between those layers. It provides a reasonable starting point for decomposing an application feature into executable components and the communications required to connect them.

Vertical slices are frequently bundled together to implement larger application features and functions, while each slice continues to maintain its individual focus and separation of concerns.

The vertical slice is a useful perspective for implementing new application functionality and for modifying or re-architecting existing functionality. It is also synergistic with modern development practices such as domain-driven design, and behavior-driven development.

Domain-Bounded Context

The first challenge in service design is drawing the boundaries around each individual service component. That’s where understanding the Domain-Driven Design concept of a bounded context is very useful.

In plain English, the word context refers to the rules and conditions within which something occurs or exists. We use a bounded context to align an individual service’s functionality with functional application boundaries, rules, and data ownership.

That way the service can be developed, tested, deployed, and modified by its developers with minimal interference with, or from, other development teams. We use it to achieve the high level of granularity and separation of concerns that preserves service autonomy and independent deployability.

How do we draw domain boundaries around a context? This is not an abstract discussion of philosophy. As software developers we must decompose large complex applications into executable components with which we can build whole software systems. If we cannot identify boundaries for usable domains, we will have a hard time building composable software.

A bounded context is defined through the collaboration of the knowledgeable business users and the responsible developers. This is labor intensive and requires an iterative effort by both sides until a clear and unambiguous definition is understood and agreed upon by both sides.

Bounded contexts are the blueprints for the individual services that make up the application. The more closely they represent the desired application functionality, the more accurately and quickly the executable code can be developed.

Iteration at the context definition stage is many times less expensive than iteration at the coding and testing stage. Getting contexts right is the optimum way for business and development to collaborate — and it is the gift that keeps on giving for the life of the application.

As you will see, we will exploit the ability of the actor model to honor the context boundaries and rules — and by making each context a first-class executable component we can turn bounded contexts directly into application behaviors.

Using this perspective, a bounded context should not be a catchall for multiple application features — but rather be limited to a specific isolatable application feature and the data access it requires. A larger bounded context can aggregate those smaller contexts as building blocks to implement more complex feature sets.

The Actor Model

The actor model has been used both as a framework for a theoretical understanding of computation and as the theoretical basis for practical implementations of concurrent systems.

Actor model services are stateless and reactive, processing input messages, sending output messages, and publishing events. All messages and events can be passed through an actor’s precondition and postcondition validation, guaranteeing that assertions of required data integrity are always enforced.

Actors are close to an ideal component model for services, because:

• Actor instances are reactive and execute rules, logic, and data transformations only when reacting to a message.
• Actor instances are absolutely reentrant and stateless. They react to one message at a time and have no memory of previous messages processed. All data needed to react to a message must be in the message itself or in a persistent datastore. That means that any instance of a specific actor type can instantaneously replace any other instance of that same type — enabling the implementation of seamless failover, scaling, and load balancing.
• Actor instances pass messages to other actor instances when they need them to do something.
• Actor instances publish events when they need to tell interested subscribers about something.
• An actor instance bounded by one context can pass messages to, or publish events for, actor instances bounded by another context — enabling it to use services developed, deployed, and maintained by other teams.

Messaging

Patterns like the actor model communicate through message passing. The requester sends a message to a service or publishes an event and relies on the receiving service and its supporting infrastructure to then select and execute the appropriate logic.

Both asynchronous event messaging and synchronous request-response messaging can be implemented, giving application developers leverage to optimize communications for specific use cases and performance objectives — all within a common unifying framework.

Web Sockets is a TCP-based protocol that enables full-duplex connections between senders and receivers and can make request-response messaging effectively asynchronous.

There are advantages to applying the REST architectural style to both request-response and event messaging. A REST API is focused upon transferring state, not upon discrete application functions. All proper REST calls follow the same set of patterns to retrieve and manipulate resource state.

All RESTful messages must specify a target address. A REST request has a server address and port as a target, while an ECST event targets a queue topic. Otherwise requests and events are identical. Without a valid target a RESTful message is just a dead letter.

A request asks that something be done and an event reports that something has happened. The receiver decides what to do based upon the HTTP method of the message and its payload.

Messaging Strategies

In addition to the target, RESTful messages have a URI that identifies the resource and implementation version whose state is to be transferred or altered. Unfortunately, the URI is also sometimes used to convey a function verb, like “getUser”. That is a legitimate SOA convention, but it is not RESTful, where the HTTP method always conveys the intent as described below:

• GET a representation of the state of the specified resource instance or instances.
• POST a representation in order to create a new resource instance.
• PUT, or replace the state of a specified resource instance.
• PATCH, or modify the state of a specified resource instance.
• DELETE a specified resource instance.

A GET message may also contain an HTTP query string that can influence the selection of the instance or instances of the resource type whose representation is to be transferred.

POST, PUT, and PATCH transfer a resource representation in the body of the request.

POST requires a unique resource identifier or must create one. If a new resource identifier is generated, it is customary to return the new resource’s URL in the “Location” HTTP response header.

PUT, PATCH, and DELETE always require a unique resource identifier.

It is the responsibility of the resource implementation to know how to process the request — encapsulating the individual resource’s discrete behaviors in one place — and minimizing the number of individual API calls a developer must learn and understand.

Message passing implements loose coupling, but also can implement dynamic coupling. Dynamic coupling, using message orchestrators, provides a very powerful mechanism for implementing load balancing, failover, and dynamic scaling. Orchestrators can also be an important mechanism for implementing self-configuring systems.

Message Orchestrators

Orchestrators are the wiring that connects individual actors by organizing messaging between them and by acting as circuit breakers to mitigate cascading error conditions. Orchestrators manage the failover, scaling, and self-configuring capabilities of the actor model.

When an orchestrator starts up, it builds a map of all service resources located in its own class path, and registers its own presence with all the other reachable orchestrators on the network — exchanging maps with them. Orchestrators are federated across a network and share state information with each other. An orchestrator takes in messages addressed to a specific service version and directs them to the most performant available instance of that service version.

When services are hosted on a Jakarta EE server like Jetty, an orchestrator can be implemented as a pre-matching request filter. When a request arrives at a server that does not have an instance of the correct service and version, the orchestrator redirects the request to a server that does.

NOTE: Services can also be deployed as containerized microservices (see Designing Microservices).

Server as a Message Handler

Message orchestrators are the key ingredient necessary to deploy application components on a network without complex top-down configuration. A map of all the servers on the network and all their installed components and versions can be accessed at any time.

If a requestor has the address of any component server on the network, it can successfully send a request or publish an event to any target on the network.

Self-Configuration

Complexity is the primary limiting factor in the successful implementation of large application systems. It is the Achilles heel of large services and API management implementations.

As the number of things (APIs, services, resources) and connections between them grows, complexity increases dramatically [ c = n(n-1)/2 ]. Top-down hierarchical controls, as implemented in most systems, are ill-suited to cope with this complexity. A better solution is needed.

Many of the working machines of that complexity occur in the natural world. For the solution, we can look to self-configuring systems, the way nature copes with complexity. Self-configuring systems emerge from bottom-up interactions, unlike top-down hierarchical systems, which are not self-configuring.

Message orchestrators can be used implement dynamic coupling among federated services (which, upon start up, are cataloged by their home orchestrator). In this way control is distributed over whole systems and all parts contribute to the resulting functionality, as opposed to centralized structures that are often dependent upon a single coordinating entity.

An individual service does not need to know the network addresses of any other services with which it communicates, the message orchestrator with which it is paired is responsible for that.

This decentralized structure, inherent to self-configuring systems, gives them resiliency and robustness. When any element fails it can easily be replaced by a like element. A successful service architecture mimics the decentralized structure of organic living systems where complex capabilities can emerge from the interaction of relatively simple parts — while at the same time minimizing the complexities of configuration and deployment.

Testing

Software testing can be a difficult and time-consuming exercise under the best of circumstances. How does one test hundreds, or even thousands, of executable components that can interact across a network? The actor model has an answer — preconditions and postconditions enforced when messages are received or results computed.

Pre and post conditions validate and filter the data in the messages that actors receive and send. They can assert preconditions for incoming messages and postconditions for outgoing messages.

When an invalid state is detected, they can post an error event— which is picked up by a distributed logger, which in turn invokes an error handler if one is specified for the error type. If invoked, the specified error handler can orchestrate the cleanup or remedial actions required.

Pre and post conditions are declarative. After all, they are assertions. They can be declared for data elements, data structures, and messages. Those declarations can be used to generate executable code, and in a message-orchestrated system that executable code can be automatically invoked by an orchestrator.

When services are hosted by Jakarta EE servers, preconditions can be implemented as name bound request filters, and postconditions can be implemented as name bound response filters, which is the fundamental purpose of such filters.

This approach guarantees that the state of application messages and data match the declared rules, and that actors can always assume clean and valid input data — while effectively and continuously executing unit tests on all components wherever they are used.

Distributed loggers and error handlers ensure that error conditions occurring anywhere in the system can be reported for logging, analysis, and remediation.

This kind of testing is not a replacement for the behavioral scenario testing performed by things like Cucumber and Gherkin. It is focused only upon ensuring that data adheres to declared constraints, not that behaviors themselves are properly implemented.

The RESTful Resource

In the REST architectural pattern, requests are made to transfer the state of a resource between requesters and responders. In this model, all resources are implemented as services. These resources implement logical layers in a vertical slice. Some of those layers can be:

• Persistent data such as instances residing as a table row or rows in a SQL database, a document in a document database, or a keyed file record or records.
• State data of an electronic device or computer process.
• A façade interfacing with legacy or external APIs.
• A purpose-designed logical view representing a bounded context and using the services of one or more of the above resource types.

As one of its architectural constraints, REST leverages a layered system. This means that the requester has no need to care about where a resource version executable is located. It does not have any need to know whether it is communicating directly with a local resource or an aggregate of resources scattered across the network. It will access all of them in exactly the same way.

This allows us to directly access rows in a SQL table, documents in a document database, the state of an instrument or process, or a complex view of multiple resource types constructed to implement the bounded-context of a complex business function — all in the same way.

This provides serious advantages for how we can synchronize the persistence of related data entities within a larger application context, and how we can implement individual and multi-layered application GUI components.

User Interface

A modern Web user interface is made up of elements like scrollable tables, forms, images, and the buttons, links, scrollbars, and tabs that assist a person to use and navigate the application the UI represents. Modern Web UI frameworks can implement a programmable component model with JavaScript or TypeScript, HTML5, and CSS3.

Web Components and Contexts Implementing a Vertical Slice

Web Components provide a standard component model for the Web, allowing for encapsulation and interoperability of individual HTML elements. Web Components are supported by current versions of all major browsers.

Google Lit is a simple open source library for building fast, lightweight Web Components that are compatible with Web GUI frameworks like Angular and React, or libraries like jQuery — and can be used with either JavaScript or TypeScript.

With Lit’s tagged template literals, the browser passes the tag function an array of strings (the static portions of the template) and an array of expressions (the dynamic portions). Lit uses this to build an efficient representation of your template, so it can re-render only the parts of the template that have changed.

In this model, each Web component is a micro frontend and interacts with a server through a context service. That service implements the bounded context of a vertical slice of application functionality as a logical data view and executes both business and data access logic for the Web component. The context service, in turn uses standard REST services to access persistent data. Context services return data and, or, error messages through a standard HTTP response object.

Utilizing a logically consistent component model from the user interface, through the application business logic, and down into persistent data can help manage complexity while improving developer comprehension and productivity.

Frameworks

A framework, by necessity, comes with everything required to satisfy a wide range of projects and use cases. In practice, developing with a framework will come with features, functionality, and code that won’t be used at all within your project.

Most frameworks are built upon an underlying conceptual and architectural model. Though there are ongoing efforts to modernize them, most of today’s service frameworks are implemented with a layered Service-Oriented Architecture. This can be an impediment to implementing systems with a more modern component architecture.

A lightweight framework that implements minimal additional features can act as a middle-ground between frameworkless development strategies and older heavyweight feature-rich frameworks. Such a lightweight framework can be implemented with the newer Jakarta EE libraries.

Conclusions

Component and communications models, and development tools, that help us to isolate and focus on one thing at a time can dramatically increase our comprehension and productivity, while reducing the rate at which we make errors. That can help us to manage the essential complexity that makes software systems so hard to build and risky to develop.

To implement truly distributed application functionality requires a domain partitioned architecture with a flexible component and communications model. Such a model can distribute data and functionality across multiple servers and devices on a network while providing the flexibility and redundancy necessary to eliminate most single points of failure.

To reliably function in multiple diverse environments the model should be self-configuring for both large and small installations. It should require a minimum of technical support to deploy, configure and operate. With today’s technologies that is a practical and achievable goal.

If you found this article useful, a clap would let us know that we’re on the right track.

Thanks!

Recommended Reading

1. Composable Services, what makes a service composable.
2. The Magic of Message Orchestration, the wiring that connects composable components.
3. Designing a REST API, why transferring state representations beats remote procedure calls.
4. Is OOP Relevant Today?, why OOP is important in today’s world of distributed computing.
5. Strategic DDD by example: subdomains identification, how to define coherent and reusable components.
6. Designing Composable Services, how composable services optimize application design tradeoffs.
7. Building with Composable Services, a more detailed description of the composable services architectural pattern.