When should I use Message Queue?
First, the origin
Everything from the business architecture design and the introduction of new technologies are tricky.
Before introducing a technology, the first question that should be answered is what problem this technology solves.
Just like the micro-service layered architecture, you should first answer why you should introduce micro-services and what problems micro-services solve.
Many netizens asked, when is the use of MQ, and what kind of scene is suitable for MQ, so this article is available.
Second, what is MQ doing?
Message Queue, hereinafter referred to as MQ, is a cross-process communication mechanism for delivering messages upstream and downstream.
In the Internet architecture, MQ is a very common messaging service for upstream and downstream “logical decoupling + object understanding coupling”.
After using MQ, the upstream of the message transmission only needs to rely on MQ, and it does not depend on other services logically and physically.
Third, when do not use the message bus
Since MQ is a decoupling tool in the Internet layered architecture, isn’t it good to use MQ for all communications? This is a serious misunderstanding . The relationship between the call and the called cannot be replaced by MQ.
The shortcomings of MQ are:
1) The system is more complicated, and an MQ component is added.
2) The message delivery path is longer and the delay will increase
3) The reliability and repeatability of the message are contradictory to each other, and the message is not lost and difficult to guarantee at the same time.
4) The upstream cannot know the downstream execution result, which is very deadly.
Give a chestnut : the user logs in to the scene, the login page calls the passport service, and the execution result of the passport service directly affects the login result. Here, the “login page” and the “passport service” must use the calling relationship, and cannot use MQ communication.
In any case, remember this conclusion : the caller relies on the business scenario of the execution result in real time, please use the call instead of MQ .
Fourth, when to use MQ
Typical scenario 1: Data-driven task dependence
What is the task dependencies, For chestnuts , Internet companies often make some statistics task in the morning, there is a certain dependencies between these tasks, such as:
1) Task3 needs to use the output of task2 as input
2) Task2 needs to use the output of task1 as input
In this case, there is a task dependency between tast1, task2, and task3. Task1 must be executed first, then task2 is executed, and task3 is executed.
A common implementation for this type of demand is to use cron manual to execute the schedule:
1) task1, 0:00 execution, experience execution time is 50 minutes
2) task2, 1:00 execution (reserved 10 minutes buffer for task1), experience execution time is also 50 minutes
3) task3, 2:00 execution (reserved 10 minutes buffer for task2)
The downside of this approach is:
1) If there is a task execution time that exceeds the reserved buffer time, you will get the wrong result, because the post task is not clear whether the pre-task task is executed successfully. At this time, you must manually re-run the task, and you may need to adjust the row. Class schedule
2) The execution time of the total task is very long. Always reserve a lot of buffers. If the pre-tasks are completed ahead of time, the post-tasks will not start earlier.
3) If a task is dependent on multiple tasks, this task will be called a critical path, and the schedule is difficult to reflect the dependencies, which is easy to make mistakes.
4) If there is a task to be adjusted, the execution time of multiple tasks will be adjusted.
In any case, using the “cron schedule” method, each task is coupled, who used who hurts who knows (please comment on this method)
The optimization scheme is to use MQ decoupling:
1) Task1 starts on time, and a message “task1 done” is sent after the end.
2) Task2 subscribes to the message “task1 done”, starts the execution at the first time after receiving the message, and sends a message “task2 done” after the end.
3) the same as task3
The advantages of using MQ are:
1) No buffer is reserved, the upstream task is executed, and the downstream task is always executed at the first time.
2) Relying on multiple tasks, being handled by multiple task dependencies, only need to subscribe to related messages
3) There is a change in task execution time, and there is no need to adjust the execution time for downstream tasks.
It should be specially noted that MQ is only used to pass the message that the upstream task is completed, and is not used to deliver the real input and output data.
Typical scenario 2: upstream does not care about execution results
You need to use “call” when the upstream needs to pay attention to the execution result, and you can use MQ when the upstream does not pay attention to the execution result.
For a chestnut , many downstream cities in 58 cities need to pay attention to the “users post posts” event. For example, after recruiting users to post posts, the recruitment business should reward 58 beans. After the real estate users post posts, the real estate business should send 2 tops, second-hand users release After the post, the second-hand business should modify the user statistics.
A common implementation for this type of requirement is to use the calling relationship:
After the execution of the post publishing service is completed, the downstream recruitment business, real estate business, and second-hand business are called to complete the notification of the message, but in fact, whether the notification is normally and correctly executed, the post publishing service does not pay attention at all.
The downside of this approach is:
1) The execution time of the post release process has increased.
2) Downstream service is down, which may cause post publishing service to be affected, upstream and downstream logic + physical dependence is serious
3) Whenever you add a message that needs to know the message “post posted successfully”, the code is modified by the post publishing service. This is the most disgusting. It belongs to the typical dependency reversal in architecture design. Whoever used it hurts who knows (using Please comment on this law)
The optimization scheme is to use MQ decoupling:
1) After the post is successfully posted, send a message to MQ
The advantages of using MQ are:
1) Short upstream execution time
2) Upstream and downstream logic + object understanding coupling, except for physical connection with MQ, modules are not interdependent
3) Add a downstream message follower, no need to modify any code upstream
Typical scenario 3: The upstream focuses on the execution result, but the execution time is very long.
Sometimes the upstream needs to pay attention to the execution result, but the execution result is very long (typically calling offline processing, or calling across the public network), and often using the callback gateway + MQ to decouple.
For a chestnut , WeChat payment, cross-public network call WeChat interface, the execution time will be longer, but the caller is very concerned about the implementation results, how to play at this time?
Decoupled by the “callback gateway + MQ” scheme:
1) The caller directly calls the WeChat interface across the public network.
2) WeChat return call is successful, this does not mean that the return is successful.
3) After the WeChat execution is completed, the callback unified gateway
4) The gateway will return the result notification MQ
5) The requester receives the result notification
It should be noted here that the callback should not be called by the callback gateway to notify the result. If this is the case, each time the caller is added, the callback gateway needs to modify the code and still rely on the reverse. The callback gateway + MQsolution is used. , any new calls to WeChat payment, no need to modify the code.
Fifth. Summary
MQ is a common decoupling tool in the Internet architecture.
When not to use MQ?
Upstream real-time attention to execution results
When do you use MQ?
1) Data-driven task dependence
2) upstream does not care about multiple downstream execution results
3) Asynchronous return execution time is long
