1 hour migrations #1: SQS to GCP’s Cloud Pub/Sub

Ron Pantofaro
Jun 15, 2018 · 2 min read

Message queues play an important part in today’s modern scalable, distributed infrastructure. Amazon Web Services (AWS) offers SQS (Simple Queue Service) and Google Cloud Platform (GCP) offers Cloud Pub/Sub.

These are both managed, scalable, reliable services for publishing and consuming messages. In this post I would like to discuss in a high level how to migrate your application from using Amazon’s SQS to Google’s Pub/Sub.

Dude, where is my queue?

Before we dive a bit deeper, let’s map some terms from SQS to Pub/Sub.

While in SQS you work with queues, in Pub/Sub you work with topics and subscriptions. You’ll might ask, wait why did my queue become two separate entities? That has to do with the way Pub/Sub works. The actual queue-like paradigm in Pub/Sub is represented by a topic, but you won’t be able to do much without consumers creating and subscribing to one (or more) subscriptions for that topic.

Let’s look at the following SQS queue :

  1. Name: myqueue
  2. Region : us-east-1
  3. Message retention: 12 days
  4. Fifo: no
  5. Redrive policy : No (no dead letter queue)
  6. Maximum payload size: 256KB
  7. Message delivery delay : None

The first migration steps to Cloud Pub/Sub would be:

  1. Create a topic with the name “myqueue”
  2. Create a subscription with the name “mysubscription” for that topic.

Notice that:

  • There is no need to define a region since in Pub/Sub topics are global.
  • Message retention in Pub/Sub (at the date of publication of this post) is 7 days, so we can’t get to 12 days like the configuration we had on SQS.

Publishing messages (i.e Let them have it!)

In SQS, publishing a message is as easy as calling the SendMessage API call with as little as the queue URL and the message payload. In Cloud Pub/Sub, you can call the “publish” method with just the topic name and the payload. Easy, but just pay attention that Cloud Pub/Sub apis will return a future that you will need to wait to resolve until you can hold the sent message Id.

Receiving messages (i.e Incoming!)

To poll messages from SQS you can use the AWS api in an easy manner:

response = client.receive_message(
QueueUrl="https://somearn/myqueue",
MaxNumberOfMessages=1,
VisibilityTimeout=10,
)

Now, in Cloud Pub/Sub you can use message polling or message pushing (read more at https://cloud.google.com/pubsub/docs/subscriber).

For simplicity purposes we will use the poll mechanism, but notice I am using the asynchronous flavor:

subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(
'myproject', 'mysubscription')

def message_callback(message):
# Do something with the message here
message.ack()

subscriber.subscribe(subscription_path, callback=message_callback)

Because the polling here is non blocking, if this is the only thing the process does, you will need to have a main loop that keeps it alive.

Conclusion

In this post I covered a migration scenario from a simple SQS use to Cloud Pub/Sub. Of course there are much more complicated use cases, architectures and best practices which I’ll might cover in upcoming posts.

More reading:

Cloud Pub/Sub: https://cloud.google.com/pubsub/docs/

SQS and Cloud Pub/Sub: https://cloud.google.com/docs/compare/aws/application-services

Google Cloud Platform - Community

A collection of technical articles published or curated by Google Cloud Platform Developer Advocates. The views expressed are those of the authors and don't necessarily reflect those of Google.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade