Events on the Outside vs Events on the Inside

Back in 2005, Pet Helland has written this excellent white paper called “ Data on the Outside versus Data on the Inside “. It explores the concept of time “Then and Now” in the data. One of the key takeaways is the application of the theory of the real world to the data.

By the time you see a distant object, it may have changed!

By the time you see a message, the data may have changed!

Pet sees the data in two states i.e. Then and Now. Just by being aware of the state of the data, one can make an informed decision while producing/consuming it. Mainly it sees the data within a system ( bounded context) as a state of “now”, where data is in a highly consistent state. However, when the same data is outside the system, either shared as a response to a request or a command from an external system, it is in a state of “then”.

• The response of a request carries data as of the time response was generated from the producing system.
• A command from the external system requests a future state of the data within the consuming system.

Now State — Data On the Inside

This data is within a single bounded context, which gives it unique features and constraints as discussed in the next points.

• Transaction scope: Data being in a single bounded context, enables the read and write operations to be performed within a transaction scope.
• Consistent and Mutable: All invariants checks are made on the persisted state within a scope of a transaction scope, this means we can work on data considering it a current state and keep the state always consistent. This also allows the data to be mutable as well.

Then State — Data On the Outside

Data, when shared outside the bounded context, is considered a “then” state as it is shared outside the bounded context as well as transaction boundary. Incoming commands request a future state so they are also in a “then” state.

• Immutable / Read-Only: Once shared outside the bounded context, the data within the bounded context can change but the copy that is already shared outside is read-only.
• Grows stale with time: The other side effect is a disconnected copy of the source data, it starts growing stale with time.

The above two characteristics of the “then” state data make it a good candidate for a unique version of each copy e.g. It is more accurate to say “Helard Sun of Sunday, 7th March 2021” instead of just “Herald Sun”

Applying state on Events

When implementing an event-driven system, we have to put careful thought into what events will be produced. There are a few popular ways to identify events that will be produced:

Applying the learnings from Pet’s paper, we can categorize these events into Inside vs Outside events, sometimes referred to as Domain Events and Integration Events. Domain Events are internal to a bounded context and when generated, they live in the “Now” state of the domain until they are persisted, hence more like “Data on the Inside”. On the other hand, integration events are published outside of the bounded context, mostly for the consumption of the external systems. They start growing stale from the moment they are published to the outer world, which places them in the “Data on the Outside” camp.

Domain Events — Events on The Inside:

• Domain events are within a bounded context for inter aggregates communication
• In-memory or persisted (event sourcing)
• Consumers and producers are often in the same transaction scope.
• Part of mutable state but they are immutable themselves
• Can be lightweight: refer entities Ids within a bounded context
• Subject to faster evolution
• Event Sourcing Example
• In Process Events

Integration Events — Events on the Outside

• Inter bounded context communication
• Immutable, cacheable, read-only
• Event body is formal contract: Unique ID (aggregate id + version)
• Derived off domain events or the domain code as a side effect
• Only raised after the inner state is persisted by the aggregate.

Example

Let’s take an all-time favorite example of a shopping cart in Sales bounded context. Following are some of the key events identified by the team:

• Cart Created
• Cart Abandoned
• Cart Submitted
• Item Added to Cart
• Item Removed from Cart

Now, when this was discussed in more detail to identify commands and policies, the team discovered a business rule that prohibits essential items to be more than the allowed quantity. If a customer exceeds the limit, then items are rejected by the cart and instead, they will be added to the wish list. The wishlist will provide easy access in case the customer wants to swap it with something in the cart. This business rule is internal to the Sales bounded context.

Cart and Wishlist have close interactions and currently business has no use cases for Wishlist except for cart. With this in mind, the team has decided to keep the wishlist under the cart aggregate, therefore the same bounded context i.e. Sales. There is another bounded context that is touched upon in the above diagram, it is called ‘Order’.

The team realised that not all these events are required outside, so by applying “Data on the Outside vs Data on the Inside” on events, the team tried to identify the events that are required outside the Sales bounded context:

The interactions within the Sales bounded context can be transactional i.e. Rejecting cart items and adding a wishlist will be a part of the same transaction scope. However, generating orders when the cart is submitted will be an asynchronous operation and can be implemented as an eventually consistent state.

+-----------------------+--------+---------+
|         Event         | Inside | Outside |
+-----------------------+--------+---------+
| Card Created          | yes    | no      |
| Cart Item Rejected    | yes    | no      |
| Cart Item Added       | yes    | no      |
| Wishlist Item Added   | yes    | no      |
| Wishlist Item Removed | yes    | no      |
| Cart Submitted        | yes    | yes     |
+-----------------------+--------+---------+

Domain Events

By this categorisation of the events, we unshackle the evolution of events on the inside. Now internally these events can be implemented in a few different ways or some mixture of them:

• Inter-Aggregate in-memory/in-process events (MediatR in C#or Go Channels)
• Event Sourcing — Cart aggregate will construct the domain model from the events persisted in the local state
• CQRS — By using a message broker, cart aggregate will emit domain events and separate workers will maintain projections and/or read models by listening to the domain events.

In-Memory Events: The following diagram illustrates how in-memory event publishers and subscribers can be used by application code to use domain events to communicate:

The above diagram is an example of dotnet application using MediatR to facilitate inter-component, in-process communication. This provides flexibility to the developers to structure the code in a more modular way. Inter-aggregate domain events are a perfect fit for it. Because the publishing and consumption of events are in the same process, it allows the whole operation to be within the same transaction boundary. So if any of the subsequent event handlings fails (add wishlist), it would fail the whole operation.

Event Sourcing: Another use case of the domain events is in the event sourcing, whereby an aggregate saves to state in form of a sequence of various domain event-triggered to handle a command.

As shown in the diagram above, the cart aggregate would accept the command and then load the current state before deciding the next action. The next actions would be stored as events in the database. Again, this whole process will be within a single transaction scope.

Integration Events

To intentionally keep the example easy, the only integration event we are looking at is the “Cart Submitted”. We know that one or more external systems are interested in this event. Its “then”/”outside” characteristics mean, we should consider some of the above-mentioned constraints or guidelines. The most important attribute is its immutability and read-only nature of it. The recommendation is to provide a version of the event body.

CartSubmitted: { 
 CartId: xxxxx, 
 Version: 1, 
 Customer: xxxxx, 
 TotalAmount: $$,
 Items: [] 
}

This event would also carry the complete state to describe the event so that the consumer would not need to come back. And if for any reason the consumer application has to communicate back, then it will have the cart id and the version to refer to. This saves from the typical problem of the state being updated after the event was published. The version can also be supplied in the commands from external systems, this would allow the domain code to quickly reject the command if it is based on an out-of-date copy of the integration event.

Now if the business allows the cart to be edited event after it is submitted, then the order API knows to abandon the previous order when a new version of the cart submitted event is received. It may not be that straightforward, in case the order has since been processed already. There can be various ways to handle the situation based on the complexity of the actual business requirements, such as Workflow, Sagas, Microservices orchestration, etc. These patterns are not in the scope of this post, one can write a book explaining these in detail.

Conclusion

Event-driven systems inherit many complexities of the integration of distributed systems, by applying some of the learnings from the standard integration patterns, we can limit the complexities to the events that truly need to be exposed outside of the domain. This provides the development team more flexibility to let the domain (inside) events evolve without any consideration of side effects on the external system. Similarly, by being aware of the integration events' nature as “Data on the Outside”, we can apply some of the concepts like uniquely identifying every event (id + version). Accepting its “then” nature we make it explicit and build systems accordingly.

Originally published at http://codesimple.blog on March 14, 2021.