Integrate web application with external systems by using Message Queue
The common use case
Extra business logics will be added when a user interact with the system. For example, we want to send a confirmation email to the users after they submitted registration form, we may also want to push his information to the CRM (like salesforce) or email marketing service like Mailchimp. If we just put all the logic in one Controller, The code will be cumbersome and it will bring some bad user experiences (e.g slower server response time as many api calls are sent synchronously)
Solution & Implementation
Message queue is the key to solve the issue. Personally, I choose AWS SQS instead of other queue applications/services. Here are comparisons of different queues.
AWS SNS
In some cases, I would like to use AWS SNS + SQS to implement the solution. This implementation is called “fanout” as mentioned by AWS blog
For example, we want to send information to different services(CRM, Email marketing service, Email services) when registration completed, money transfers completed and deposits completed. We can create different SNS topics and subscribe different SQS queues to these topics. Thus, we can save api calls in our main application:
If we use SQS only, we need to send 3(services) * 3 (steps completed)= 9 api calls to AWS.
If we use SNS + SQS, we need 3(pushing messages) api calls to SNS.
However, if sending information to different services is not compulsory (e.g. sending information to Mailchimp is optional to your user as you sell it as a feature) I prefer to use SQS only.
My demo
Source code, It is implemented by Symfony3.
Tips and findings
SQS Queue Types
There are two types of message queues. The Standard queue and FIFO (first in first out) queue. Standard queue provides unlimited throughput but no order is guaranteed. FIFO guarantees the order of the message but with limited throughput.
SQS Queue feature
The standard queue is distributed, thus sometimes something unusual happens. For example, if there are only 10 messages in the queue.
You may only get 1 or 2 messages even you set 10 to the MaxNumberOfMessages parameter when you polls messages out of SQS. However, if the volume of messages is large enough (e.g. 10,000 messages in the queue), you will get 10 messages if you set 10 to the MaxNumberOfMessages parameter.
SQS Long Polling
In short, enable long polling will save your cost. see http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html.
To enable long polling, just pass a non-zero integer (maximum 10) to WaitTimeSeconds parameter when you send requests to SQS.
Supervisord
Supervisord is used to auto-restart the long-running worker (for my case, it is long running PHP command).
The first time, I get an issue…
When you meet error like Could not dispatch event … it must be the wrong permission of supervisord.log and supervisord.pid. I simply put those two files into project dir and it solves the problem.
Cache
I use Redis as a cache to store the “last_restart_date”. “last_restart_date” is used to safely stop a long running background daemon/worker. This is inspired by Laravel Queue
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —
Edit 8/2018:
Symfony 4.1+ ships with a new Messenger Component, see my new post to see how easy it is to use Messenger Component + AWS SQS to deal with asynchronous messaging!
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —
Edit 12/2019
It has been a long time since I wrote the first version of this post and Symfony has polished the Messenger component a lot since 4.1. Moreover, Symfony and Symfony Cast now have better/more comprehensive documents/video tutorials about Messenger Components. I would strongly recommend you read/watch those materials.