Message Queue: A Complete Guide
What are Message Queue?
Message queue is one critical component in your software architecture design especially when we start moving our legacy system from monolithic to microservice architecture. Message queue is in essence a queue data structure which you could put your message in and take your message out but order not assured(depends on your configuration).
There are several concepts in message queue:
- Message: it is the data that is transferred between services and put in and taken out from the queue. Message could have several formats like JSON, String, and Plain Text.
- Producer service: it is the service generate message and put them into the queue. Some examples of producer service like ordering service, transaction service etc.
- Consumer service: it is the service to take message out of the queue and process them. Some examples of consumer services like storage service, email notification service etc.
Some Advantages of Message Queue
Introducing message queue in microservice architecture could bring several advantages.
Firstly, Message queue could make sure communication between services in an asynchronous way, improving user experience. Imagine a scenario where you make a transaction in an application, and you get notification until your transaction(ordering, payment) all completed. This definitely takes a lot of time, and ruins user experience. But with the help of message queue, users just make an order, pay and leave. These information will be put into message queue and be processed latter.
Secondly, Message queue helps decouples components in your microservice architecture. Therefore your microservices servers are independent and could scale up horizontally, and have high level fault tolerance. For example, if one of your server crash others will work seamlessly. And all the heavy weight work server could scale up independently.
Some Popular Message Queue Broker System
There are several message queue broker systems in the market. One is basic queue model like RabbitMQ, AWS SQS etc. The other is publisher/subscriber model such as Kafka, AWS SNS etc.
Basic queue model is very easy to understand. There is one queue and producer put message in the queue, and consumers take message from the queue in a competitive manner(image you have three or more consumers waiting to consume message from one queue). But the basic queue model will have an advantage that it will make sure all the message are processed in order.
Pub/Sub model is more loose coupling than basic queue model. Publishers and Subscribers are decoupled from each other by Brokers. Broker is actually a working node in a message queue cluster. And in pub/sub model you could have many topics you want in a broker such as order topic, storage topic etc. But subscribers should subscribe to specific topic in order to receive messages from it. In order to increase reliability and fault-tolerance, there are several brokers in a message queue cluster and for each broker there are several topics in it. And for each topic there are several partitions across brokers. However, there is only one leader partition for each topic. If this leader partition fails, other partition will be re-selected as leader to make sure system reliable. And Kafka is a typical pub/sub model message queue and has high efficiency in processing streaming data but it cannot make sure process messages of a topic in order.
