1 hour migrations #1: SQS to GCP’s Cloud Pub/Sub

Message queues play an important part in today’s modern scalable, distributed infrastructure. Amazon Web Services (AWS) offers SQS (Simple Queue Service) and Google Cloud Platform (GCP) offers Cloud Pub/Sub.

These are both managed, scalable, reliable services for publishing and consuming messages. In this post I would like to discuss in a high level how to migrate your application from using Amazon’s SQS to Google’s Pub/Sub.

Dude, where is my queue?

Before we dive a bit deeper, let’s map some terms from SQS to Pub/Sub.

While in SQS you work with queues, in Pub/Sub you work with topics and subscriptions. You’ll might ask, wait why did my queue become two separate entities? That has to do with the way Pub/Sub works. The actual queue-like paradigm in Pub/Sub is represented by a topic, but you won’t be able to do much without consumers creating and subscribing to one (or more) subscriptions for that topic.

Let’s look at the following SQS queue :

1. Name: myqueue
2. Region : us-east-1
3. Message retention: 12 days
4. Fifo: no
5. Redrive policy : No (no dead letter queue)
6. Maximum payload size: 256KB
7. Message delivery delay : None

The first migration steps to Cloud Pub/Sub would be:

1. Create a topic with the name “myqueue”
2. Create a subscription with the name “mysubscription” for that topic.

Notice that:

• There is no need to define a region since in Pub/Sub topics are global.
• Message retention in Pub/Sub (at the date of publication of this post) is 7 days, so we can’t get to 12 days like the configuration we had on SQS.

Publishing messages (i.e Let them have it!)

In SQS, publishing a message is as easy as calling the SendMessage API call with as little as the queue URL and the message payload. In Cloud Pub/Sub, you can call the “publish” method with just the topic name and the payload. Easy, but just pay attention that Cloud Pub/Sub apis will return a future that you will need to wait to resolve until you can hold the sent message Id.

Receiving messages (i.e Incoming!)

To poll messages from SQS you can use the AWS api in an easy manner:

response = client.receive_message(
    QueueUrl="https://somearn/myqueue",
    MaxNumberOfMessages=1,
    VisibilityTimeout=10,
)

Now, in Cloud Pub/Sub you can use message polling or message pushing (read more at https://cloud.google.com/pubsub/docs/subscriber).

For simplicity purposes we will use the poll mechanism, but notice I am using the asynchronous flavor:

subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(
    'myproject', 'mysubscription')

def message_callback(message):
    # Do something with the message here
    message.ack()

subscriber.subscribe(subscription_path, callback=message_callback)

Because the polling here is non blocking, if this is the only thing the process does, you will need to have a main loop that keeps it alive.

Conclusion

In this post I covered a migration scenario from a simple SQS use to Cloud Pub/Sub. Of course there are much more complicated use cases, architectures and best practices which I’ll might cover in upcoming posts.

More reading:

Cloud Pub/Sub: https://cloud.google.com/pubsub/docs/

SQS and Cloud Pub/Sub: https://cloud.google.com/docs/compare/aws/application-services