Customized queue and retry mechanism with Google Cloud PubSub using Go

Have you ever had two microservices where they communicated with each other using RestAPI, but due to an infrastructure error, (system down or traffic was high) they failed to hit. And as a result, the data between the two services were out of sync.

Then, what is the solution? Read more on how I make amends and successfully fix the condition:

First, we need to create a system that works for queuing & retry. The system’s job is only to accommodate incoming requests from the sourceSystem into a queue. Then the request queue at the front, one by one is consumed and hit to the destinationSystem.

But it turns out that a message from the QueueRetrysystem failed to process the request to the destinationSource, then what to do?

For every request that fails to hit, we will store it in the last queue, the goal is for the retry process to occur. Meanwhile, requests that successfully hit the destinationSource are removed from the queue.

Can it really be? What are the tools?

I used Google Cloud PubSub by using Ack & nAck. The following is an overview of the queue & retry system that I implemented at Warung Pintar.

The issue that I’ve experienced while in production, namely the Subscriber/Consumer doesn’t work, so unACKed messages pile up, making the graph go up like the following picture, the solution is to look for the error log and fix it, if you can’t find it, then you just need to restart the QueueRetrysystem.

Graphics of unACKed messages that are piling up.

In the next story, I will break down the details into 3 parts:
1. Ordering
2. Re-deliver
3. Implementation in Go

Stay tuned!