Switching from SQS to Google Pub/Sub
At ShareIQ we’ve used a mix of different cloud providers over the past two years. Eventually it seemed that Google’s offering was the most comfortable to use, so we currently have all our micro services running on Google, but for legacy reasons we’re still using AWS SQS for messages.
Replacing Queues
Instead of queues, Pub/Sub has topics and subscriptions. You publish a message to a topic and it’s delivered once per subscription name. E.g. on topic Hello you can have subscriptions World and Everyone. Since we’re looking to replace AWS, we decided to use the name of a queue as the topic name as well as the subscription name. Now we can have multiple worker instances running and one of them will take care of a message. The result looks kind of like this:
const PubSub = require('@google-cloud/pubsub')function getSubscription (name) {
const pubSub = PubSub(config)
return pubSub.topic(name).get({autoCreate: true})
.then(topic => topic.subscription(name).get({autoCreate: true})
}
Thanks to autoCreate, we don’t even need a setup step to create the topic and subscription. If none exist with this name, they will just be created for you.
Receiving Messages
Pub/Sub offers options to pull and push strategies for receiving messages. Since we have a high throughput of sometimes hundreds of messages per second in a single instance, we’d like to use the pull strategy. For SQS we’ve written some logic that’s polling messages and keeping track how many messages are currently in flight. With a subscription we can start receiving messages by adding an event listener:
const subscription = getSubscription('MyName')subscription.on('data', (message) => {…})
To Be Clarified
We haven’t fully transitioned yet, so there are still open points
- Marking messages as done
- Setting a limit of how many messages should be processed in parallel
- Publishing messages
- Adding metadata to messages
- Sending delayed messages
So far it seems that most of the SQS feature set we were using can be easily replaced by Pub/Sub.
