MuleSoft Integration Patterns — Reliability Pattern

Published in

Another Integration Blog

7 min readJan 5, 2024

In the second article of this series, we will explore the Reliability Pattern and its implementation in MuleSoft with RabbitMQ. We will learn how we can use it to ensure the secure and reliable delivery of data between systems, even in the face of network or database failures.

Reliability Pattern — How it Works

The image above shows a Reliability Pattern model where an API is responsible to receive and send data to a given target system. Upon validation and acceptance of the data, it is encapsulated into a message and sent to a queue. The client then receives a 202 Accepted HTTP status response, allowing it to close the connection. The message remains in the queue until another processor retrieves, processes, and successfully sends it to the target system. If delivery fails after several retries, the message is moved to a Dead-Letter-Queue (DLQ) for special error handling, alerting, and manual processing. This ensures that the main queue only contains potentially validated data for further processing.

Utilizing resilient, transactional queues prevents the loss of state in the event of system failures. Additionally, they foster decoupling between different components of the system, facilitating easy scalability when needed.

When to Use the Reliability Pattern

The Reliability Pattern is suitable for scenarios that allow asynchronous processing, demand zero tolerance for message loss, and require guaranteed data delivery. For example, a fintech startup processing high volumes of online payments daily through data exchanges with internal and external systems would benefit from the Reliability Pattern to enhance reliability and resilience.

Implementing the Reliability Pattern with MuleSoft and RabbitMQ

Business Case — Online Ticket Selling System

Let's build a Ticket Selling API demo to demonstrate the Reliability Pattern. This type of system should be designed to serve massive events, ensuring the best possible user experience while processing a high volume of requests.

Before we start, here you can find the source code.

Now, let's design our demo API with two main asynchronous steps:

Ticket Buy Order Reception: The system receives the order, confirms receipt to the client immediately, and places the order in the main queue.
Ticket Buy Order Processing: The system asynchronously gets the order and validates it against business rules. Successful validations result in an email confirmation, while invalid orders are sent to another department for handling. The entire process is this demo, will be implemented in a single MuleSoft application for simplicity.

Setting Up RabbitMQ

Let’s use a cloud RabbitMQ service through the CloudAMQP.

Go to https://www.cloudamqp.com/ and create a free account.
Then, create a new Instance using the plan “Little Lemuer”. It is free and enough for our purposes.
Copy the username and password by clicking on name of the plan you created. This will be used to connect to the RabbitMQ service from the MuleSoft app.

4. Now, click on. “RabbitMQ Manager”. You should see the RabbitMQ manager as below:

5. Now, we will need to create two Exchanges and two Queues. Name exchanges as EXCHANGE.TICKETS and EXCHANGE.TICKETS.DLQ. The two queues should be named TICKET.PURCHASE and TICKET.PURCHASE.DLQ.

6. Now we need to create the bindings. They associate the Exchanges with the Queues and the main queue with its Dead-Letter-Queue.

Binding configuration for the DLQ queue — Binding Configuration for the DLQ queue.

7. At the end of the steps above, when clicking in the Queues and Streams menu, you should have the following:

TIP: You can use the RabbitMQ queue screen to create a message, send it, and purge queues. This is useful to test whether the DLQ binding is working correctly.

Congrats! Everything is set on the RabbitMQ service!

Creating MuleSoft app

Now, create a new project in Mulesoft.
Import the dependency AMQP connector from the exchange.
Create a flow named ticket-purchase-flow.

Make it listen in http://localhost:8081/tickets/purchase to receive the following JSON POST:

{
"userId" : 112623,
"quantity" : 1,
"unitPrice" : 50.00,
"customerName" : "Mr. Anderson",
"customerEmail": "neo@gmail.com"
}

The flow will receive the POST, create a message into RabbitMQ, and will return an HTTP 202 status code to the API client.

Create the AMQP Configuration to connect with RabbitMQ with the configuration you got in the first the section “Setting up Rabbit MQ”.

Configure the AMQP Publish component as below:

4. Create a flow named ticket-processing-flow.

This flow should manually trigger the processing of one message at a time at the queue while logging in the console the steps. Make it listen for a POST to http://localhost:8081/tickets/process.

If the processing is successful, it will send an “Ack” to RabbitMQ completely removing the message from the queue and sending an e-mail with “Purchase Completed” confirmation. If a validation error happens, it will send a “Reject” to RabbitMQ. In this case, since we have a DLQ associated with this main queue, the rejected message will be automatically moved to the associated DLQ for later processing.

Configure the AMQP Consume component property as:

Configure the AMQP Ack and Reject components AckId property as:

5. Create the subflow “validate-message-subflow”.
This flow simulates a business restriction that will make the purchasing order invalid. In this case, passing a quantity more than 5 in the request JSON will raise an error.

6. Finally create the flow “dlq-handler-flow”.
This is a continuous flow that listen to the DLQ on new messages. Process it by sending it to another API for further processing (could also generate an alert via e-mail, etc.).

Configure the AMQP Listener as:

Running the app

Case 1: Success Purchase

Use Postman, to set a POST request to http://localhost:8081/tickets/purchase with the following JSON body:

{
"userId" : 112623,
"quantity" : 1,
"unitPrice" : 50.00,
"customerName" : "Mr. Anderson",
"customerEmail": "neo@gmail.com"
}

You should see a successful log and a message stored in the TICKET.PURCHASE.

Now let’s process this message. Use Postman, to set a POST request to http://localhost:8081/tickets/process with an empty body.

You should see your log confirmation and see no message in the queue.

{
  "status": "success",
  "statusDescription": "Ticket Purchase completed successfully!",
  "data": {
    "userId": 112623,
    "quantity": 1,
    "unitPrice": 50,
    "customerName": "Mr. Anderson",
    "customerEmail": "neo@gmail.com",
    "id": "219aa37d-951f-47e1-a0ba-d4a8db4b0339",
    "ackId": "1-2-cda82aff-c6ce-4361-912e-edcdbfaa441e"
  }
}

Case 2: Invalid Purchase

Use Postman, to set a POST request to http://localhost:8081/tickets/purchase with the following JSON body. (Note the quantity was changed to 6):

{
"userId" : 112623,
"quantity" : 6,
"unitPrice" : 50.00,
"customerName" : "Mr. Anderson",
"customerEmail": "neo@gmail.com"
}

Now let’s process this message. Use Postman, to send a POST request to http://localhost:8081/tickets/process with an empty body.

You should see your 2 log confirmations. The first from the processing flow and the second from the DLQ flow.

{
  "message": "Bad Request",
  "errorCode": "400",
  "errorDetails": "ERROR: Quantity cannot be more than 5. This message was moved to the DLQ.",
  "data": {
    "userId": 112623,
    "quantity": 6,
    "unitPrice": 50,
    "customerName": "Mr. Anderson",
    "customerEmail": "neo@gmail.com",
    "id": "b89a03a8-95e5-4109-871e-ddcebb59026d",
    "ackId": "1-2-e6df2e10-13cb-4c82-a395-b43a0c4a8c82"
  }
}


{
  "message": "DLQ Message sent to a Process API (simulation)",
  "data": {
    "userId": 112623,
    "quantity": 6,
    "unitPrice": 50,
    "customerName": "Mr. Anderson",
    "customerEmail": "neo@gmail.com",
    "id": "b89a03a8-95e5-4109-871e-ddcebb59026d",
    "ackId": "1-1-e36f63cb-143c-4090-b93c-f8b078106363"
  }
}

When the message is Rejected by the processing flow it is moved immediately (by RabbitMQ) to the DLQ. At this point the DLQ handler flow, which is a continuous listening process, detects a new message and processes it.

If you found this demo interesting, you can explore it further by:

Implementing a retry mechanism before rejecting messages.
Learning about transaction options within the Try scope when working with the AMQP Listener in Manual or Auto Ack modes.

Summary

This article explained the Reliability Pattern with a demo that showcases how MuleSoft and RabbitMQ can be combined to build a reliable Ticket Selling API.

Happy coding!