Failsafe: building more reliable serverless applications
How to set up a dead-letter for an Eventarc trigger using gcloud CLI and track your serverless exceptions
Welcome to the first of a 2-part series on how you can set up dead-letter policies for Eventarc triggers on serverless applications. This time we look at creating one using gcloud commands. Next time, we will guide you on doing the same using Terraform.
Serverless has presented a big opportunity in The Telegraph’s quest for more efficient and environmentally-friendly services. Instead of a server or Kubernetes-based microservice running 24 hours a day, we can leverage GCP (Google Cloud Platform) Cloud Functions that only execute code for a small fraction of that time. One of the crucial pieces of building out a reliable serverless-based service is how and when you pull the trigger on the code executions and what happens when the code throws an exception.
Enter Eventarc triggers, which offer breadth and depth. Breadth in what can act as a trigger. Do you need to run code based on logging events, a pubsub topic, write events on a bucket, or changes to specific file or filetypes? They can all be triggers. And depth in what they can execute. Eventarc triggers can fire up Cloud Run containers or Cloud Functions, giving you the choice of container-based or serverless code executions that are suitable for a great deal of tasks.
But like many out-of-the-box solutions, Eventarc triggers are also a closed box with little control over them. In this blog, we’ll explore how to configure the Eventarc trigger’s behaviour in a way that gives us more control.
At its core, Eventarc events consist of a simple pubsub architecture. If a trigger is set to fire on a bucket object write event, it would essentially send a message to its pubsub topic, and a subscriber would consume the message and act on it.
Every pubsub subscription can be configured to have a dead-letter policy. Messages are sent to a dead-letter topic if a number of delivery attempts are exhausted with no acknowledgement from the subscription. This provides us with a way of exposing and managing failures of the code to complete.
Below, we will show you how to create your dead-letter policy using gcloud CLI.
The initial setup
Prerequisites
To perform the following steps, you will need to ensure the following:
- You have the gcloud CLI installed.
- You have a GCP account and have configured its access to a GCP project with gcloud CLI.
- Your account must have the requisite permissions to create the resources detailed below in your project.
Getting started
In thefunction-deploy branch of this repository, you will find instructions on deploying a simple cloud function. The cloud function is set to trigger a pubsub message to a topic and will print a greeting message containing the message.
If the message="fail", the code is set to raise an exception, which will cause the function to exit ungracefully.
Create the pubsub topic and function
## create pubsub topic
gcloud pubsub topics create demo-pubsub-topic
## create cloud function
gcloud functions deploy demo-pubsub-function \
--gen2 \
--runtime=python311 \
--region=europe-west2 \
--source=./src \
--entry-point=subscribe \
--trigger-topic=demo-pubsub-topicTrigger function (print greeting)
gcloud pubsub topics publish demo-pubsub-topic --message="World"This will cause the function to execute and print the greeting Hello World, and we can verify this by looking at function logs and running the following command.
gcloud beta functions logs read demo-pubsub-function — region=europe-west2 — gen2Trigger function (fail)
gcloud pubsub topics publish demo-pubsub-topic --message="fail"This message will cause the function code to throw an exception, which will cause the function to exit ungracefully. As before, we can verify this by checking the function logs.
Enable retry
At this point, the function is set to not retry upon failure. We can change that by adding a — retry flag to our function deploy command:
gcloud functions deploy demo-pubsub-function \
--gen2 \
--runtime=python311 \
--region=europe-west2 \
--source=./src \
--entry-point=subscribe \
--trigger-topic=demo-pubsub-topic \
--retryIf you now open your GCP console and check the function information, and go to the trigger tab, you’d see that the retry-on-failure is enabled.
Also, if you head over to the Pubsub page and click on the topic we created, you will see that it has a phantom pubsub subscription attached to it. We never created that subscription; instead, the gcloud functions deploy command took the liberty of creating and attaching that subscription to the topic.
By clicking on the subscription and then the Details tab, you’d see that the dead-lettering is disabled on that, but back-off and retry are enabled because we added the--retry flag.
Dead-letter Setup
First thing we’d need is a pair of pubsub topic and subscription to use as dead-letter. A dead-letter topic is just another pubsub topic.
Create a dead-letter topic/subscription
The following commands will get us the pubsub topic and subscription.
gcloud pubsub topics create demo-deadletter-topic
gcloud pubsub subscriptions create demo-deadletter-subscription --topic=demo-deadletter-topicAttaching the dead-letter topic
From the UI, it’s easy for us to get the Eventarc subscription name, but where’s the fun in that? We will instead use the gcloud command to get the subscription id and then use that. Since it’s not straightforward, we’d have to fetch this information in two different steps:
- Get the function info and grab the trigger name from that
- Using the trigger name, get trigger info which contains the subscription name
The following commands will help us achieve that:
trigger_name=$(gcloud functions describe demo-pubsub-function --region=europe-west2 --format='value(eventTrigger.trigger)')
trigger_subscription=$(gcloud eventarc triggers describe $trigger_name --location europe-west2 --format='value(transport.pubsub.subscription)')We can run echo $trigger_subscription to confirm whether we have the correct subscription id.
Update subscription to attach the dead-letter
Let's update the Eventarc subscription and attach the dead-letter topic to it.
gcloud pubsub subscriptions update $trigger_subscription \
--dead-letter-topic=demo-deadletter-topicGrant required permissions to the project service account
One last and necessary step is to give the GCP project’s pubsub service account the roles/pubsub.subscribe permission on the Eventarc subscription and roles/pubsub.publisher to the dead-letter publish. This is very important and it will ensure the project service account has all the permissions required to be able to read the failed messages and forward them to the dead-letter topic.
The project’s pubsub service account email format is as below:service-<GCP_PROJECT_NUMBER>@gcp-sa-pubsub.iam.gserviceaccount.com
For example, if your project number is 1234567890, the service account would be service-1234567890@gcp-sa-pubsub.iam.gserviceaccount.com
We can give the service account the required permissions by running:
## assign pubsub subscriber role to read messages from trigger subscription
gcloud pubsub subscriptions add-iam-policy-binding $trigger_subscription \
--member="serviceAccount:service-<GCP_PROJECT_NUMBER>@gcp-sa-pubsub.iam.gserviceaccount.com" \
--role="roles/pubsub.subscriber"
## assign pubsub publisher role to allow publishing the dead letter topic
gcloud pubsub topics add-iam-policy-binding demo-deadletter-topic \
--member="serviceAccount:service-<GCP_PROJECT_NUMBER>@gcp-sa-pubsub.iam.gserviceaccount.com" \
--role="roles/pubsub.publisher"That’s it, we’re done! We can confirm that by checking our Eventarc subscription.
We can also confirm that the entire setup is configured correctly by checking the DEAD LETTERING tab.
Test the dead-letter functionality
Send a message to the pubsub topic that will make the function code raise an exception.
gcloud pubsub topics publish demo-pubsub-topic --message="fail"Check pubsub dead-letter for failed messages
Due to the retry and backoff mechanism, this might take a couple of minutes to come through.
gcloud pubsub subscriptions pull demo-deadletter-subscriptionThe final code can be found in the main branch of this repo.
Stay tuned for part 2 of our guide, where we will walk you through how to configure a dead-letter to Eventarc using Terraform
Irfan Durrani is a lead platform engineer at The Telegraph. You can follow him on LinkedIn. You can also follow The Telegraph Engineering team on LinkedIn.
