Dead lettering in Google Cloud Pub/Sub

Prerequisites:

You should know about Google Cloud Platform and have basic idea about the Pub/Sub, Google Cloud’s fully managed service for real-time messaging.

Intent:

To introduce the setup of dead lettering capability for subscription/s in Pub/Sub.

Not to discuss the importance and implementation of dead letter queue/dead letter messaging in the world of microservices.

With that level set, let’s begin:

Simplified view of how Pub/Sub works:

A publisher is someone (mostly a service), who wants to publish an information about an event.

It publishes the information to a Topic as a message, for which there could be many subscribers.

The subscribers (Mostly a service) can either pull the new message or it will be automatically pushed to them, based on their type of subscription.

The subscriber acknowledges the message indicating that it has received the message. The subscribers can even decide to process the message first and then acknowledge.

Consider a scenario where you have a Pub/Sub topic created in a Google Cloud project and the subscription for which is in a different Google Cloud project. (This is a pretty common scenario.)

What is a dead letter topic?

Dead letter topic is another topic that you can create, to forward the message if a subscriber fails to acknowledge the message after certain delivery attempts.

You can have something like below:

This can be very useful in case of spikes or sudden increase in loads on the systems.

Below are few scenarios where if the service which is consuming the messages from main subscription (called MainSub above) is:

a) Having performance/availability issues and failing to acknowledge the message

b) Having performance/availability issues and failing to successfully process the message and hence not acknowledging the message

c) Doesn’t want to process the message because the service/dependent services are not ready

In case of failures from the service to acknowledge the message after predefined delivery attempts, the message will be forwarded to a dead letter topic (called DeadLetter above). The same service; when alive and kicking; can pull the messages from dead letter topic as well to see what it missed during the times of unavailability.

Another implementation could be service1 using subscription ‘MainSub’ doesn’t want to process the messages with faulty values and hence doesn’t acknowledge the message. Such messages will be forwarded to ‘DeadLetter’ topic and service2 using ‘DeadLetterSub’ can feed these messages and/or statistics about them back to the originator to improve the input.

There are other possible and better ways to make use of dead letter messages, but they are not the topic of discussion of this article. The motive is to describe the steps to setup dead lettering in Google Cloud Pub/Sub.

Let’s begin the core discussion about enabling dead lettering for a subscription in Pub/Sub:

When you create subscription for a topic in Google Cloud Console, you get the option to enable the dead lettering.

When you click the checkbox you are asked to specify the topic:

It makes sense to have a topic different than the source topic to be the dead letter topic. Still if you set the original topic as a dead letter topic, by mistake or on purpose, you’ll see warning like this:

After you enable the dead lettering and create the subscription, you’ll see a new tab called DEAD LETTERING like below for the subscription:

It states the requirements for dead lettering to work as expected.

• You need to select an active dead letter topic.
• The dead letter topic should be different than the source topic.
• The dead letter topic should have an active subscription.
• The service account used by Pub/Sub should have publisher role to publish the messages to dead letter topic.
• The service account used by Pub/Sub should have subscriber role to forward the messages from current subscription for which dead lettering is to be activated.

You can create default subscription for the dead letter topic by the option available in console itself.

If you have adequate rights, you can grant the publisher and subscriber roles to the service account.

Clicking the “GRANT PUBLISHER ROLE” runs below command in background.

gcloud pubsub topics add-iam-policy-binding dead-letter-topic-id \

— member=”serviceAccount:$PUBSUB_SERVICE_ACCOUNT”\

— role=”roles/pubsub.publisher”

Clicking the “GRANT SUBSCRIBER ROLE” runs below command in background.

gcloud pubsub subscriptions add-iam-policy-binding subscription-id \

— member=”serviceAccount:$PUBSUB_SERVICE_ACCOUNT”\

— role=”roles/pubsub.subscriber”

Where PUBSUB_SERVICE_ACCOUNT is the service account used by the Pub/Sub for the project where subscription is being created.

It has the format service-{project-number}@gcp-sa-pubsub.iam.gserviceaccount.com and below is an example from IAM page of the Google Cloud project console.

Upon successful creation of subscription for the topic and granting publisher/subscriber roles, it should be all good like in the below image.

That’s it, just these simple steps and you have setup the dead lettering topic for your Cloud Pub/Sub topic.

Below are the reference pages which give more information:

https://cloud.google.com/pubsub/docs/subscriber

https://cloud.google.com/pubsub/docs/dead-letter-topics#gcloud