The event notification pattern

Learn how the event notification pattern can help you build event-driven systems and what pitfalls you need to be aware of.

Photo by Kristina Tripkovic on Unsplash

In the second blog of the ‘engineers guide to event-driven architectures’ series I discussed key concepts to designing and implementing a successful event-driven architecture. One of those is building knowledge on commonly used event driven software design patterns.

And there’s plenty design patterns to discuss, but it seems fitting to start with the design pattern that is commonly present in event-driven systems: the event notification pattern. Let’s explore how using this pattern impacts the design of events, communication between components, and very importantly, which tradeoffs it comes with and some potential pitfalls to look out for.

How event notification pattern works

With the event notification pattern, events are used to notify other microservices in the system that an interesting change has occurred. The notification event is small and concise as it only contains a reference to the state that was changed. This is likely some unique identifier, supported with some helpful metadata that can help consumers determine if the change is relevant to their responsibility.

To give an example of helpful metadata, rather than implementing a range of separate events for different changes a customer can make to their profile, you could have one customerProfileUpdated event with a field that specifies which profile attributes changed. This would allow a consumer listening for those events, but only being interested in address changes, to decide if and how to respond to the event. Be aware that this does have the potential to reduce clarity on intent on a code level, so it’s a tradeoff.

A notification event for a customer placing an order might look like this (modelled using the cloudevents.io standard):

Using an event broker combined with the publish/subscribe messaging model to publish events like the one above allows the system to respond with the appropriate response. This is what that might look like in a simple commerce system when sticking with our order placement example (detailed description of steps below visual):

Using event notification pattern in a product ordering flow

This is what happens when an order is placed in a microservice architecture that uses the event notification pattern:

1. In this system, the billing and shipping microservices need to be aware the moment a new order is placed. Since we are using publish/subscribe, these microservices will subscribe for orderPlaced events at the event broker.
2. When a customer places an order, the front-end application sends a https request to the order microservice with the order details. Those details are stored by the order microservice in its database.
3. The order microservice notifies other microservices in the system by producing an orderPlaced event to the event broker. All microservices that have subscribed to that event type will now be able to consume the event that order microservice just produced.
4. Both the billing and shipping microservice consume the event. The event, like our previous example, contains only the order identifier, and the source which indicates where the order details can be retrieved from.
5. Now that the billing and shipping microservices are aware of the new order, they do a callback to the https endpoint of the order microservice to retrieve the details on the order, so they can perform subsequent processing.

When comparing using the event notification pattern to a purely RESTful microservice architecture, there are two key benefits:

• Better maintainability through natural coupling
• Complexity reduction in event producers

Better maintainability through natural coupling

A key benefit the pattern brings is the ability to reverse the direction of dependency between the order microservice and other microservices in the system using the publish/subscribe messaging model facilitated by the event broker. It’s no longer the order microservice’s responsibilities to explicitly inform the billing and shipping microservice and every microservices that might be added in the future, that cares about orders.

It emits that fact as an event using the publish/subscribe model and that’s it. It neither knows nor cares about what components are interested. This leads to a much more natural and manageable level of coupling in the system as introducing additional services that need to know about orders be done without changing the order microservice.

Complexity reduction in event producers

In addition to more natural coupling, the complexity of the order microservice is also greatly reduced because explicit awareness about other domains and components in the system is no longer needed. Also, the responsibility to ensure downstream consumers receive the notification shifts from the order microservice towards the event broker with it publish/subscribe messaging. This further reduces complexity within the order microservice.

So given these two rather large benefits, using event notification pattern is definitely worth considering for most microservice architectures that use RESTful communication.

Now, let’s also explore some potential pitfalls that I feel you should be aware of when it comes to this pattern.

The consistency over availability trade-off

For anyone building a distributed system, CAP theorem is a massively important concept I explained in the first blog of the ‘engineers guide to event-driven architectures’ series. Any distributed system is forced to make a trade-off, and the event notification pattern favours to have consistency over availability. Which can be either a benefit or a drawback, depending on the challenges that the system is attempting to solve.

When using the event notification pattern data replication is minimal as state is not present in the events. As state is only available through a callback to the producer there is clear data ownership.

By not replicating data throughout the system challenges like eventual consistency, data replica convergence and data conflict resolution are side stepped. This is a massive benefit, as these are particularly challenging to truly get right. However, it does come at the cost of availability.

Since consumers have to perform a callback to get state the producer needs to be available at all times or the consumer cannot perform its task in response the event. Going back to our previous example, the billing and shipping of an item cannot occur if the order microservice cannot provide the order details. These dependencies can be one of the biggest enemies of distributed architectures.

Again, the trade-off that the event notification pattern makes can either be the right choice for the challenges that a system is attempting to solve or it’s not. It’s a topic of fierce debate in the community which is “better”, or the lesser evil if you will in terms of overall complexity:

• Favour consistency over availability and implementing resilience measures to deal with the headache of microservice dependencies and minimize their impact on overall system availability.
• Favour availability over consistency guarantees by replicating data allowing microservices to operate autonomously, at the cost of introducing eventual consistency, data replica convergence and data conflict resolution challenges into the system.

I do not have the answer, but I feel that one key consideration is how well a team is equipped to deal with the complexities that mentioned above on data replication impacting consistency vs having runtime dependencies impacting availability. In general, implementing resilience measures on API endpoints to maximize availability and limit the impact of disruptions is a relatively well-known practice for teams that have been building APIs which is a part of many modern systems. For them, the event notification pattern is a good pattern with which they can be successful in production in the short term because the availability challenge is fairly well-known and understood.

Callback pressure

Besides the previously discussed availability impact, there is one-more potentially quite significant drawback to the callback towards the producer. And this is the potential pressure it puts on the producer. If an event is consumed by nine consumers, that’s nine incoming calls to the producer which likely occur in quick succession. There are also operational scenarios in which that pressure can spike further. For example, if one of those consumers was down for maintenance and is “catching up” processing all events that accumulated during that period.

The producer either needs to be dimensioned properly to manage that pressure, but in some cases, like with legacy and systems that are hard(er) to scale, it’s not that easy. In that case rate-limiting or other protection is likely needed, which solves the backing system from tipping over, but it does slow down consumers considerably. If you have requirements for near real-time outcomes where’s there’s legacy systems involved that you don’t control and tip over when you look at them funny, event notification pattern might not be the best fit.

Event processing

With any communication in a distributed system, there’s always the risk of network failures, and events are not immune to this problem. which is why most event brokers offer at-least-once delivery semantics as the default behaviour. If you are unfamiliar with these concepts, have a look at my previous blog where it’s discussed.

Because of network failures, messages might be resent, and therefore consumed multiple times by a consumer. So, events have to either be idempotent, meaning you can process them multiple times without state becoming invalid, or consumers need to keep track of previously consumed events.

Another potential issue with processing events is out of order processing. The majority of event notification pattern implementations I’ve seen over the years produce an event at the (successful) conclusion of a business process or action. In those cases, out of order processing is not a concern.

When related events are produced throughout a business process or action you will want to ensure events that the messages are processed in the right order. The most common solution for ordered processing is to use an event broker that supports FIFO and apply a partitioned consumption pattern instead of a competing consumers setup. If you are not familiar with these concepts, they are covered in my previous blog. An alternative is implementing a robust reordering strategy inside the consumers. To be warned, this does get complex quickly, since time in a distributed system is a challenge in itself.

The complexity trap of event flows

As Martin Fowler fairly warns, the simplicity of event notification pattern can be a trap when complexity in the system increases and logical flows emerge that run over various event notifications. It’s harder to observe and understand, debug and modify such a flow when compared to looking at a single piece of code. Designing maintainable event flows can be challenging, so be sure to learn about common patterns to use. My colleague Xavyr Rademaker recently published a nice article on this topic which is definitely worth a read.

So, that concludes the most important pitfalls of using the event notification pattern, let’s wrap it up.

Conclusion

The event notification pattern is by far the most common pattern you will find in event-driven architectures. It provides benefits to common communication patterns and is a relatively approachable pattern for teams that are getting into event-driven architectures.

The biggest thing to be aware of is the fact that the pattern’s favours consistency over availability, since state is not included in the event. So, if availability guarantees and (near) real-time processing are what you are looking for, this is not the pattern you are looking for. If you did not get the Star Wars reference, I am officially disappointed.

That concludes the third addition to the ‘engineers guide to event-driven architectures’ blogging series. Stay tuned for the next one, where I will discuss event carried state transfer pattern, and what benefits that brings to the table. Until the next blog, I wish you an awesome day and happy coding!