Why we use Publish-Subscribe pattern over Observer pattern

Publish-subscribe pattern has been around for sometime now and provides easy scalability and loose coupling. When building a system it is important to have a good design in place to avoid sleepless nights as the system grows. It is important to spend some time thinking over it rather than just jumping in to code.

Working at a startup time is a luxury that I never have. So when it came to data APIs for frontend I followed the traditional client server pattern. On every request I calculated all the data and sent it out as response. This worked fine in the beginning but as the users and their data increased the API responses started getting slower. This was a red flag and something had to be done before it blew up in my face.

I started off by breaking up the things that were being done in API calls in two categories:

1. Essential — those tasks without which the user cannot proceed
2. Background — those tasks which can be run in the background and the user does not have to wait for it to complete.

Let us consider what happens after a successful payment. Our payment gateway vendor sends a response. User is waiting to see the success screen. At a very basic level the following things are being done:

• Make database updates (essential)
• Perform related calculations (background)
• Send email and SMS notification (background)

Thankfully all of the APIs catering to the UI on all platforms are written in Node.js so making this change was not a difficult job. Now for all the APIs as soon as the essential tasks are over, a response is returned to the UI and the remaining tasks continue in the background. This worked fine for a while but I knew it had two issues:

• Code was tightly coupled
• This approach would not do well at scale

However making this change gave me some time to think over a better approach to solving this problem.

I reviewed the things that needed to be done tech wise at a company level and not just on user actions. Goalwise being an investment platform needs to do a lot of number crunching at a daily level and most of these calculations depend on the price of investments at the end of each day. Rather than breaking up every API task in essential and background it made more sense to classify events that occur in the company. I began with the simplest and most important business unit:

• Change of transaction status — A customer has made a purchase or a redeem transaction and its status has changed

How it works

Every investment transaction in Goalwise goes through the following phases in its life cycle:

Initiated State → Intermediate State → End State

On every change of state the publisher (in this case the node API mentioned earlier) after completing the essential tasks publishes a message to AWS-SNS. SNS in turn triggers lambda functions which perform the relevant background tasks.

Rough representation of transaction handling architecture

This design has given me room to perform a lot of tasks in the background without consuming the resources serving the data APIs. It has also given me all the flexibility that comes with using lambda.

Conclusion

• Having a long term vision and working towards it is important. I couldn’t allocate all the time in building the current architecture so I broke it up in phases.
• It is important to keep scale in mind when coding.
• Always leave room for enhancements.