Implementing an event driven architecture with RabbitMQ
An event driven architecture is often used as a way of communication between decoupled services and it is common pattern in many modern applications. In this architecture, a service publishes an event when something notable happens, such as when it updates a business entity, and other services subscribe to those events. When the subscriber receives an event it can update its own business entities, which might lead to more events being published.
The problem
Even though some of our services here at Simply Wall St made use of message queues, which is a way to implement an event driven architecture, they were often isolated. Some legacy application would use their own message queue solution where producers and consumers would only live in a project and be isolated from anything else. Notifying other systems that something happened was often a cumbersome task.
In some cases configuration values from other projects needed to be imported to bypass this limitation. In other cases, services would skip message queues all together and implement synchronous HTTP calls to other services directly. The later would add network latency and more points of failure to the task being performed.
The Pub/Sub model
The Pub/Sub model allows a publisher to notify all subscribers that something has happened in the system.
In this scenario the producer doesn’t really care much about the response or what happens after it publishes a message. It’s up to the consumers to perform their own tasks when a message is received.
This approach seemed to solve our problem:
a) Systems would fire a message when something important happened, e.g. a company’s analysis data has been updated.
b) Any subscriber interested in that event could listen to that message and decide what to do with it.
The ideal scenario
Instead of isolated message queues with multiple flavours (We had Kafka, Disque and RabbitMQ), which have little portability across projects the use of a centralised Pub/Sub model would be more beneficial.
Bringing this to the Simply Wall St ecosystem, an example of this implementation can be seen below:
The producer, in the example above called Company analysis worker, can publish to a topic (a message queue router) that the data for Company X has been updated. The message router will distribute the same message to all services subscribed to topic. Each consumer is now responsible for make use of that message and scale individually to handle the load.
Consumers and producers will use a centralised message queue broker instead of isolated message queues, increasing the portability portability across projects.
How did we get there?
The first step towards the adoption of an Event Driven Architecture was choosing a message broker to be used across all projects. Popular technologies that allow for the implementation of this model are RabbitMQ and Kafka.
It’s out of the scope of this article to do an extensive comparison between these two technologies but below are the main reason we decided to go with RabbitMQ instead of Kafka:
- Our messages needed to be consumed and removed. We din’t need to store the messages indefinitely or keep a transaction log;
- We didn’t want every application to keep an offset of the messages consumed. That would be an additional complexity that is not needed for our use case.
- We needed support of delayed message queues (fire a message to be consumed in X minutes, hours or days in the future).
The main takeaway here is not that one is better than the other. RabbitMQ was the best option for our problem. Kafka is also an amazing technology but had way more features than we needed so we’ve decided not to deal with the extra complexity.
After the decision was made the team dedicated some time to create internal SDKs for our two main backend languages (Typescript/Node.js and PHP) to be used across all projects leveraging two amazing open source projects, amqplib and php-amqplib.
Next Steps
With the foundations in place the team now has the means to quickly plug into the main events happening in the SWS ecosystem. Customer facing features like our new watchlist and internal caching mechanisms are already taking advantage of this new architecture.
Our focus now is in consolidating all the legacy systems into RabbitMQ which we intend to write more about in the future.
