Aggregated message processing system

In my last blog, I shared an use case about processing stream of data through server less compute using AWS Lambda. This time I am going to talk about a an architecture to process data coming from multiple streams (SNS) and processing them through SQS. I used this architecture to build a scoring model for the users participating in our program. Share with me if you have used similar mechanism in your use cases.

System Architecture

Aggregated Message Processing Architecture

Architecture Break Down

For simplicity lets say a product sold by the user goes through different sections in the manufacturing supply chain (including various validations like quality check etc.,). We want to collect certain information and derive some conclusions or make sense out of all relevant notifications pertaining to the user. As per the diagram above, say product information comes from different sources or process paths from different systems (Notification #1,#2 and #3, etc.,). I plugged SQS queues onto these notifications and processed them and feed them to the scoring model to build user scores. Let’s see how,

Subscribing Queue to SNS

In my last blog I talked about using Lambda for listening to SNS notifications. But Lambda cannot be used in this use case because as part of message processing we have to make some service calls within our network. So I used SQS to subscribe to SNS notifications and processed messages. Here’s how to subscribe SQS queues to SNS topics.

Subscribing to SNS topic

SQS Long polling

In our service I built individual processes that polls these queues through SQS’s long polling (a single process for multiple queues might cause your application to become blocked from processing available messages in the other queues while waiting for a queue that doesn’t have any available messages) and receive messages continuously from these queues and process them. You can read about the long polling below,

Summary

I was able to build a highly scalable message processing system that processes 20 million messages a day published from different notification streams. However this time it involved writing code/configurations to associate processes with the queues and to process them. Details of the SQS API used to receive message can be found here,


Feel free to share your use cases and stint with AWS components. Also please share/critique your comments about the blog.