Switching from SQS to Google Pub/Sub

Johannes Scharlach
Jul 10, 2017 · 2 min read

At ShareIQ we’ve used a mix of different cloud providers over the past two years. Eventually it seemed that Google’s offering was the most comfortable to use, so we currently have all our micro services running on Google, but for legacy reasons we’re still using AWS SQS for messages.

Replacing Queues

Instead of queues, Pub/Sub has topics and subscriptions. You publish a message to a topic and it’s delivered once per subscription name. E.g. on topic Hello you can have subscriptions World and Everyone. Since we’re looking to replace AWS, we decided to use the name of a queue as the topic name as well as the subscription name. Now we can have multiple worker instances running and one of them will take care of a message. The result looks kind of like this:

const PubSub = require('@google-cloud/pubsub')function getSubscription (name) {
const pubSub = PubSub(config)
return pubSub.topic(name).get({autoCreate: true})
.then(topic => topic.subscription(name).get({autoCreate: true})
}

Thanks to autoCreate, we don’t even need a setup step to create the topic and subscription. If none exist with this name, they will just be created for you.

Receiving Messages

Pub/Sub offers options to pull and push strategies for receiving messages. Since we have a high throughput of sometimes hundreds of messages per second in a single instance, we’d like to use the pull strategy. For SQS we’ve written some logic that’s polling messages and keeping track how many messages are currently in flight. With a subscription we can start receiving messages by adding an event listener:

const subscription = getSubscription('MyName')subscription.on('data', (message) => {…})

To Be Clarified

We haven’t fully transitioned yet, so there are still open points

  • Marking messages as done
  • Setting a limit of how many messages should be processed in parallel
  • Publishing messages
  • Adding metadata to messages
  • Sending delayed messages

So far it seems that most of the SQS feature set we were using can be easily replaced by Pub/Sub.

Johannes Scharlach

Written by

I write about #nodejs, #serverless and startups. Love tech and math.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade