Developing Message-Based Applications with RabbitMQ
Build a distributed food delivery application with Node.js and RabbitMQ. Learn how message-based asynchronous communication works in a distributed system.
Introduction
Most modern applications follow one of two thriving architectures — Monolithic or Microservices. In a Monolithic architecture, the application is structured as a single entity making it quite easy to develop, test and deploy. This is a good choice for small applications that do not anticipate a lot of changes. However, for large applications, a Monolithic architecture has proven to be difficult to maintain. The components that constitute a Monolithic architecture are tightly coupled. A small change introduced in one can severely affect the application, if not appropriately implemented. Microservices, on the contrary, segregate the various functionalities of an application into standalone services. These services interact with each other using network calls that follow RPC, REST, Message-based or a combination of these communication styles depending on whether they require synchronous or asynchronous communication.
This article aims to explain the significance of asynchronous communication in a distributed system and how it can be facilitated with the help of RabbitMQ, a very popular open-source message queuing software. We shall build a system that requires asynchronous communication to process various tasks in the backend — more precisely, a distributed message-based food delivery application using Node.js, Express, MongoDB, RabbitMQ and Docker. We’ll also be injecting faults in the system to examine the resilience it offers.
Request-Response Systems
The significance of message-based systems can be comprehended by analysing some of the drawbacks of a request-response system such as REST (Representational State Transfer) or RPC (Remote Procedure Call). Request-Response systems employ synchronous communication. In such a system, one service (client) calls another service (server) directly — often through a load balancer — using a protocol such as HTTP.
Each service is associated with a URL or an IP address that other services use to initiate a connection. As the Microservices architecture comprise numerous independent services, the number of connections between these services become colossally large; consequently, convoluting the system. Additionally, after making the request, the client has to wait for a response from the server. In other words, the client is blocked from performing other tasks until it receives a response or times out. A client triggering an inherently time-consuming service such as one which requires significant computing resources can exacerbate this situation. This can impact the user experience if the client is a frontend application as the UI becomes unresponsive.
Message-Based Systems
Message-based systems work differently. In such a system, rather than allowing two applications/services to directly communicate with each other, an additional component is introduced as an intermediary. This additional component is known as the Message Broker. The Message Broker is configured to route a message from the sender to the intended receiver, as shown in img-2.
A message is just a piece of information that is sent between two applications. Introducing a Message Broker provides some key advantages. It aids in decoupling applications. Since applications interact by employing a Message Broker, they can communicate without directly connecting to each other. Each application only needs to adhere to the data format of this Message Broker and does not need to handle the nuances of interacting with different services. This simplifies incorporating heterogenous applications in the system. Additionally, it provides the functionality of queueing messages which is immensely helpful when triggering a compute intensive service with a significant request processing time. The sender no longer remains blocked from executing other tasks as it no longer waits for a response. Adding messages onto an intermediary queue ensures that a request is not lost even if the receiver is unavailable. This enhances the reliability of the system. Queueing also aids in handling load spikes, permitting the receiving services to be scaled proportionately without losing any messages. From a much broader perspective, Message Brokers increase the flexibility of the system.
However, it is crucial to not overlook the fact that the sending service does not get an appropriate response as opposed to a request-response system. The sender can remain confident that its request has been received and can continue processing other tasks without getting blocked. Consequently, the communication is asynchronous and is often compared with email since the sender does not expect an immediate response. Another point to note is that the system is now immensely dependent on the Message Broker. If the Message Broker goes down, the system is compromised. This is resolved by deploying a cluster of Message Brokers.
RabbitMQ
Overview
RabbitMQ is an extremely popular open-source Message Broker that is used for building message-based systems. The key elements that constitute RabbitMQ — Producers, Consumers, Exchanges, Bindings and Queues — are shown in img-3.
The applications that generate messages are called Producers. These messages are published to the RabbitMQ Exchange. The Exchange, depending on its configuration, forwards the messages to one or more Queues which act as buffers for storing messages. The link between the Exchange and the Queue is called Binding. The applications that consume from the Queues are called Consumers. It is important to note that once the Producer publishes a message to the Exchange, it does not wait for a response i.e, it is not blocked. This form of communication is facilitated by the Advanced Message Queueing Protocol or AMQP.
Advanced Message Queueing Protocol (AMQP)
Though RabbitMQ supports several messaging protocols, the most popular one is the Advanced Message Queueing Protocol (AMQP). AMQP is a binary protocol. Contrary to text protocols such as HTTP, where the information exchanged between the client and the server is in human-readable text format, AMQP exchanges information (messages) in a binary format. Instead of relying on different fields, the participating entities agree upon a certain structure and find the required information at specific positions in the transmitted binary data.
The binary data is transmitted as frames, as shown in img-3. The first byte stores the type of the frame and can assume the value HEADER, METHOD, BODY or HEARTBEAT. The next byte stores the channel which identifies an independent thread of messages. Although the client establishes only a single TCP connection with the Message Broker, the connection is multiplexed — contrary to HTTP/1.1. This means that the client and the message-broker can utilise the same TCP connection to transmit multiple independent threads of messages. The next byte indicates the size of the Payload. The first three bytes are collectively called the frame header. The Payload holds data corresponding to the type of the frame. For example, if the type is METHOD, the payload will contain the name of the operation to be performed. If the type is BODY, it will contain the message data that the sender is transmitting.
Exchange Types
RabbitMQ Exchanges can be configured in four different ways depending on how the messages are to be routed — Direct, Fanout, Topic and Header. As explaining the different types of Exchanges would require an article of its own, this article will cover the Fanout Exchange. In a Fanout Exchange, messages are forwarded to every queue that is bound to the exchange. This is particularly useful for consumers that need to process the same message in different ways — such as in an online ordering platform where one queue can be used for generating an invoice PDF and another one for sending the order details to the seller.
Burgernaut
Overview
The application that we’ll be building in this article is called Burgernaut. Burgernaut is a food delivery platform that allows customers to place food orders and track them — well, virtually. It basically mimics order processing and delivery with timeouts. The order placed by a customer is forwarded to an available restaurant and a confirmation email is sent back to the customer. The customer also receives an order ID that can be used to track the order.
The application has been developed with RabbitMQ, Node.js, Express, MongoDB and Docker. Its purpose is to demonstrate how a simple message-based system can be built. This would not resemble an actual large-scale system. In fact, this is more suitable for an application that triggers a long running task and emails the details to the client.
System Architecture and Flow
Burgernaut consists of three integral components — order-service, restaurant-service and email-service, as shown in img-6. The order-service exposes REST endpoints that allow clients to fetch the food menu, place an order and track the order in real-time. Once the client places an order, the order-service persists the order details on a MongoDB server, publishes it to a RabbitMQ Exchange and returns a confirmation response which includes the order ID to track the order. The Exchange is configured with a fanout pattern which publishes the order to the two queues that are bound to it - order.process and order.confirmation. While placing it on the queue, the ‘status’ field of the order holds the value ‘pending’. A restaurant-service consumes the order from the order.process queue and an email-service consumes the order from the order.confirmation queue. Once the restaurant-service consumes the order, it modifies the ‘state’ of the order to ‘accepted’ in the database. The email-service on the other hand sends an order confirmation to the email address specified in the order. After a pre-defined time period, the restaurant-service modifies the ‘state’ to ‘delivered’ in the database indicating that the order has been delivered.
Burgernaut uses a message-based architecture to enhance scalability, flexibility and loose coupling among the various services it comprises. RabbitMQ, by default, ensures that all the messages consumed from a queue are distributed in a round-robin fashion. Since the restaurant-service and email-service consume from queues, they can be independently scaled - though physically, scaling out the restaurant-service would mean opening new restaurants. The services are configured to process only a limited number of orders at a time to mimic actual business since restaurants have limited staff. The remaining orders wait on the queue to be consumed. The restaurant-service sends an ‘acknowledge’ to RabbitMQ once the order is fully processed - delivered, in our case - to prevent it from being added back onto the queue. If one of the restaurants stop functioning say, due to a power loss, RabbitMQ, being aware that the TCP connection between the restaurant-service and the Exchange has been closed, would automatically add any order that was being processed back onto the queue, making it ready to be consumed by another restaurant-service. This enhances the fault-tolerance of the system. The number of orders that can be handled by the restaurant-service and the amount of time it takes to process an order can be configured with their respective environment variables at the time of deployment.
As is evident, the system relies on asynchronous communication for its components. This is difficult to achieve with REST or RPC based systems. The system is also dependent on a highly-available and fault-tolerant queueing service for queueing the orders. This prevents overwhelming the downstream components during peak load intervals. RabbitMQ aids in satisfying these requirements. It also enhances the user experience. Instead of rejecting an order during peak load intervals or downtime, the order gets queued and a response is sent immediately to the client. As soon as a restaurant is available, it will consume the order. Simply put, it tells the client — “Hey, we got your order and are working on it. Here’s an order confirmation ID to track it.”
Developing Burgernaut
I took the liberty of adding all the three services — order-service, restaurant-service and email-service in the same repository 🤓 Go ahead and clone the application from my GitHub repository. As far as RabbitMQ is concerned, we only need to focus on the mqService.js file present in the src/services folder in each of the three application folders. The rest of the code involves creating an Express application, configuring routes and connecting, querying and sending data to MongoDB.
The mqService.js file present in order-service/src/services represents a Producer and consequently contains the code to connect to the exchange and publish a message as shown below:
The application uses the amqplib library to communicate with RabbitMQ. The amqpConnect() function is used to establish a TCP connection with the exchange and create a Channel. As explained in the AMQP section of this article, the TCP connection is multiplexed with the help of Channels. The publishOrderToExchange() function simply invokes the publish() function on the channel object to publish a message — the JSON request payload, in our case — to the exchange. The injectExchangeService() is an express middleware that injects the publishOrderToExchange() function in the request object. This allows it to be easily extended in the future.
The mqService.js file present in email-service/src/services and restaurant-service/src/service represents a Consumer and consequently contains the code to connect to the exchange and consume a message. The code for restaurant-service is shown below:
There are a few important differences here. The code binds a queue to the exchange and configures the prefetch — which sets the maximum number of messages that can be processed at a time. When a message arrives, processOrder() is invoked. This is defined in orderController.js present in restaurant-service/src/service/controllers.This method updates the status of the order in MongoDB to ‘accepted’. Once the order is processed — mimicked with a setTimeout() — the status is updated to ‘delivered’ and the ack() function on the channel object is invoked to tell the Exchange that the order (message) has been processed successfully. Failing to invoke ack() will place the message back onto the queue.
Running Burgernaut
Burgernaut consists of containerized services that are deployed using Docker Compose. If you’re new to Docker, I recommend reading my article — ‘Containerize your Personal Projects with Docker’.
Docker Compose deploys each service in a Docker Network, thereby creating a virtual distributed environment. The docker-compose.yml file is shown below. Modify the EMAIL_ID and EMAIL_PWD environment variables in email-service to send email confirmations. In the restaurant-service, control the maximum number of orders that can be processed simultaneously using PREFETCH_COUNT and the order processing time using ORDER_DELIVERY_TIME:
To prevent building the images at the time of deployment with docker-compose up, build the images first using the command:
$ docker-compose buildTo replicate the architecture shown in img-6, use the command given below:
$ docker-compose up --scale restaurant-service=2Observe the services (containers) being created:
Docker Compose launched one container of MongoDB, RabbitMQ, order-service, email-service and two containers of restaurant-service. Use the Postman collection present in the /test folder to interact with the application.
Placing an Order
Placing an order with place-order request defined in the Postman collection will produce a response similar to the one shown below. The order ID is also highlighted. Ensure to update the email address present in the body of the request to receive order confirmations.
Tracking the order with this order ID using the track-order request will show the status as accepted if it is within the ORDER_DELIVERY_TIME defined in the docker-compose.yml file.
Tracking the order past the ORDER_DELIVERY_TIME will show the status as delivered.
Also, check your inbox for the confirmation — if you had specified your email address in the payload of place-order request.
Understanding the Asynchronous Tasks
Let’s examine the logs to understand the asynchronous tasks in the backend for the order — 6004942dedbd3a00125a9aea.
- The order-service_1 container receives a POST request. It immediately responds with a status code of 201 and the payload shown in img-8.
- The Exchange ‘fans-out’ the order to both the queues.
- The restaurant-service_2 container accepts the order from the
order.processqueue and starts processing the order. - The email-service_1 container accepts the order from
order.confirmationqueue and send outs an email to dhanushnarayan1991@gmail.com as shown in img-11. - order-service_1 gets a request for tracking the order and responds with a status of 200 and the payload shown in img-9.
- restaurant-service_2 delivers the order when the timeout defined by ORDER_DELIVERY_TIME finishes.
Placing an order again will route it to the other restaurant service - in this case, restaurant-service_1. This is because RabbitMQ dispatches messages from a queue in a Round-Robin fashion.
As an exercise, I encourage you to try placing more than 2*PREFETCH_COUNT orders —assuming that there are 2 restaurant-service containers — and observe how the orders get queued up.
Fault Injection
Scenario 1
Let’s kill restaurant-service while an order is being prepared (i.e, during the timeout) and observe how RabbitMQ gracefully sends the unprocessed order to another restaurant-service.
Place an order with the Postman request and execute the command given below in a new terminal window before the order gets ‘delivered’. Use the container id of the restaurant-service processing the order.
$ docker container kill <CONTAINER-ID>
The logs shown in img-13 reveal that the restaurant-service_1 container was killed while processing the order. The TCP connection between this container and RabbitMQ was terminated before it could process the order and send an ‘acknowledge’ — with ack(). Hence, RabbitMQ placed the unprocessed order back onto the queue allowing restaurant-service_2 to consume and process it.
Scenario 2
To understand how orders get queued, let’s kill all the restaurant-service containers and place multiple orders. Kill the containers using the command shown below:
docker-compose up --scale restaurant-service=0 -dNow place the orders and track them using Postman. With no restaurant-service to fulfil them, all the orders will be shown as ‘pending’.
Spin up two containers of restaurant-service and observe what happens.
docker-compose up --scale restaurant-service=2 -d
Recall that the RabbitMQ Exchange was publishing messages to queues. As the restaurant-service containers were unavailable to consume from the order.process queue, the messages remained on the queue. When two restaurant-service containers were spun up, the orders were instantly dispatched to them. This is the beauty of a message-based architecture! The system can handle the unavailability of consumers by queueing up the messages. Had the system been implemented with a request-response architecture, all the orders placed during the downtime would’ve been lost. For an actual firm, this would unfortunately result in a loss of revenue.
Conclusion
Message-based communication is an integral part of distributed systems. They allow heterogeneous components to seamlessly interact over the network. We saw how developing message-based systems with RabbitMQ facilitated the integration of asynchronous tasks such as sending an email and triggering a time-consuming process. We also injected two common faults and observed how RabbitMQ handles them. In our system, the client immediately receives a response with an identifier that allows it to track the task while the task is being executed asynchronously in the backend. This is just one example of a message-based architecture. We’ve barely scratched the surface!
You can find the complete source code in my GitHub repository.
Further Reading
- RabbitMQ tutorials — https://www.rabbitmq.com/getstarted.html
- A good explanation on the types of exchanges — https://www.cloudamqp.com/blog/2015-09-03-part4-rabbitmq-for-beginners-exchanges-routing-keys-bindings.html
- REST, RPC and Brokered Messaging — https://medium.com/@natemurthy/rest-rpc-and-brokered-messaging-b775aeb0db3
- Monolithic vs Microservices Architecture — https://articles.microservices.com/monolithic-vs-microservices-architecture-5c4848858f59
- Messaging in Microservices -https://microservices.io/patterns/communication-style/messaging.html
