Schedule Cloud Functions with Cloud Tasks (for Document TTL) Part 1
Ready to step into the future of automation? Let’s dive into the world where Cloud Tasks and Firestore TTL (Time to Live) blend together, simplifying your work and boosting your productivity. This journey is all about unlocking the potential of Google Cloud’s tools to automate tasks, manage your Firestore documents more effectively, and ensure timely execution of functions. Let’s get started on this exciting journey together.
This article is divided into 2 parts, and in this first part we will make Firestore document TTL along with the step by step setting up and understanding each and every little thing while also taking care of the security issues. In the 2nd part we will build an event scheduling system that automatically notifies users according to time scheduled on event status changes.
Hey Reader: The article can be long because there’s a lot to learn, so take your coffee and let’s get started 😚.
Breaking Free from Polling Limitations
Periodic polling can only take us so far. Sure you could rely on scheduled functions to run at regular intervals, scanning through a custom-built scheduler to determine if any pending tasks are due. But here’s the thing — it comes with a host of downsides that might leave you yearning for a more elegant and efficient solution.
- First, you have to invest valuable time and effort in constructing that scheduler, meticulously describing the work that needs to be done when. It’s an additional layer of complexity that could be avoided.
- Second, scheduled functions have their limitations, they can execute as frequently as once per minute, which means you might experience delays of up to a minute — a delay that might be simply unacceptable for time-sensitive operations.
And let’s not forget about unnecessary overhead. With periodic polling, you end up paying for repeated functions invocations and scheduler queries that might ultimately yield no actual work to be done. It’s an inefficient use of resources.
- Picture a reservation system that grants users a strict 15-minute window to complete a reservation. Once that elapses, the system needs to terminate the incomplete reservation and free up it for others.
- Or imagine a thrilling game that challenges players to conquer a level within 2.5 minutes, enforcing a time limit to prevent cheating.
- And what about a lively quiz game that demands lightning-fast answers within a 10-second countdown before moving to the next question.
For these scenarios and more, relying on scheduled functions just won’t cut it. You need a solution that is cost-effective and flexible, capable of delivering the precision and control your application demands.
That’s where the Cloud Tasks comes in
By leveraging Cloud Tasks, you can efficiently schedule the invocation of a “callback” function exactly when follow-up work needs to be executed. It’s a smarter, more streamlined approach that eliminates the need for unnecessary repetition and empowers you to optimize both cost and performance.
The process of Cloud Tasks is simple and straightforward. Begin by creating a queue that suits your needs, and if desired customize its configuration to optimize performance. Once your queue is set up, it’s time to add Tasks using the provided SDK. Each task can be tailored to your requirements, allowing you to define an HTTP request with a payload that precisely outlines the work that needs to be executed. Think of it as a future callback mechanism, where the designated HTTP function eagerly awaits instructions on what action to take when the scheduled time arrives.
Note: Make sure to check out Cloud Task limitations and quota limits for better understanding with Cloud Tasks.
Cloud Tasks working flow
I’ll assume you are already familiar with the Cloud Functions and Typescript for better understanding the code we’re going to write for scheduling. Here’s one diagram for clarification on how Cloud Tasks is going to work with Cloud Functions.
Set up Cloud Tasks
Cloud Tasks require billing before using it in your project. So you need to complete the billing process and then you will be allowed to use Cloud Tasks in your project. Here is the breakdown of the pricing of Cloud Tasks (it’s cheap, you can schedule 1 million tasks for free every month).
First of all, Enable the Cloud Tasks API in the Cloud Console.
You can follow the documentation to create queue. You will also need to have the gcloud CLI installed also configured with your Firebase project.
You just need this line to run and create a named queue. I will name the queue “my-scheduler”:
$ gcloud tasks queues create my-schedulerIf gcloud gives you a warning, you can ignore it as long as your output statement includes the queue was created.
Start with writing functions
With Cloud Tasks set up, you can now use it in your function code. Cloud Tasks provides a client SDK for node so install it with npm by running the command below.
$ npm install @google-cloud/tasksNow, let’s import all of the Firebase modules as well as the newly added @google-cloud/tasks module. And make sure to import it with the JavaScript require syntax.
import * as functions from 'firebase-functions'
import * as admin from 'firebase-admin'
const { CloudTasksClient } = require('@google-cloud/tasks')
admin.initializeApp()With this done now first we will cover the first scenario by making a Firestore document TTL and then to tackle the event scheduling system. Let’s face it.
Hey Reader: You might be tired, make sure you take a break and get fresh before diving into actual code 😀.
Firestore document TTL using Cloud Tasks and Cloud Functions.
Let’s start by defining an Firebase Firestore onCreate Trigger function that will be invoked when a document is created in for example “events” collection.
export const onCreateEvent = functions.firestore.document("/events/{eventId}").onCreate(async snapshot => {
// here goes the scheduling code.
});Now, we will look at when the user wants to expire the document. Having this in mind we will introduce two additional fields that will specify the expiration time of the document, these fields can be your document fields if you are implementing the same scenario in your app. expireSeconds and expireTime will be two fields where expiresSeconds will store the current time in seconds and expiresTime will store exact current time in timestamp. So will express these two methods inside the onCreateEvent function.
const data = snapshot.data() as { expireSeconds: number, expireTime: admin.firestore.timestamp };
const { expireSeconds, expireTime } = data;Next, I will store the expiration time in seconds in a variable and some conditions accordingly.
// This is local variable for storing expirationAtSeconds
let expirationAtSeconds: number | undefined
// Extract expiration time based on provided fields
if(expireSeconds) {
if(expireSeconds > 0) {
expirationAtSeconds = Date.now() / 100 + expireSeconds;
}
} else if (expireTime) {
expirationAtSeconds = expireTime.seconds;
}
// If not time is set for the document we will return
if(expirationAtSeconds == undefined) {
return;
}After having the expirationAtSeconds let’s do some configuration for the task queue.
const project = json.parse(process.env.FIREBASE_CONFIG!).projectId;
const location = ‘us-central1’;
const queue = ‘my-scheduler’;Now with the help of Cloud Task SDK provide an acceptable path to the queue.
const taskClient = new CloudTaskClient();
const queuePath: string = taskClient.queuePath(project, location, queue);Last but not the least, let’s finally configure the URL to the callback function and a payload to deliver, and one last thing the name for the task which will later help us in canceling the task.
const url = `https://${location}-${project}.cloudfunctions.net/schedulerCallback?eventId=${snapshot.id}`;
const payload = { snapshot.id };
const taskName = `projects/${project}/locations/${location}/queues/${queue}/tasks`;You might be thinking why the taskName is so long, and why only this ? can’t it be an arbitrary or simple name?
Well, creating cloud tasks with names must follow a hierarchical naming scheme which allows cloud tasks to uniquely identify tasks across multiple projects, locations and queues. You can create tasks with arbitrary or simple names but it would create a conflict and make it impossible for Cloud Tasks to determine which task to execute.
By requiring that tasks names include project ID, location, and queue, Cloud Tasks ensures that each task has a unique name and can be executed without conflict.
Now, let’s build an configuration for the Cloud Task:
const task = {
name: `${taskName}/myTask-${snapshot.id}-${expireSeconds}`,
httpRequest: {
httpMethod: ‘POST’,
url: url,
body: Buffer.from(JSON.stringify(payload)).toString(‘base64’),
headers: { ‘Content-Type’: ‘application/json’ }
},
scheduleTime: {
seconds: expirationAtSeconds
}
}To the task configuration, we provide:
- Unique name: which contains snapshot.id or eventId and expireSeconds to be unique from all the other tasks in the queue.
- scheduleTime: for scheduling the task that will later execute at the provided expirationAtSeconds.
- httpRequest: we make an HTTP request to the given url and pass the JSON content body with the contents of the payload.
Note: the encoding base64 is just required by Cloud Task API. The function will still receive the raw stringified JSON.
Finally, I will just enqueue the task to the queue.
await tasksClient.createTask({ parent: queuePath, task })And… we’re done with the first cloud function onCreateEvent.
Now we need to write an HTTP callback function that would be invoked by Cloud Tasks at the right time.
export const schedulerCallback = functions.https.onRequest(async (req, res) => {
// accessing the eventId that we sent over HTTP POST request as a query param.
const eventId = req.query.eventId as string;
try {
await admin.firestore().collection(“events”).doc(eventId).delete();
} catch(e) {
console.log(e);
res.status(500).send(e);
}
});In the function above we pull that eventId that we sent as a query param in the http request. And using that eventId we delete that document from the events collection. Make sure you include both expireTime and expireSeconds fields in your cloud firestore document according to the requirements of your project.
Finally: we are done with our first job by making a firestore document TTL. It was easy, right? I knew that 😀.
Canceling scheduled tasks
This can also be one scenario, if the user wants to cancel the already scheduled task. It is also pretty simple, let’s face it.
- Include one more field like expireTime and expireSeconds. And name it expirationStatus This must be of type string.
- Use Firebase Cloud Firestore onUpdate trigger to look for expirationStatus and accordingly delete/remove the task.
If the expirationStatus task was updated to “canceled”, use the queue path to cancel the task.
As I said earlier, to include expireTime and expireSeconds fields in your cloud firestore document, make sure you include expirationStatus also according to your project requirements. And from your app simply call the regular update method of Cloud Firestore on the event document to update the expirationStatus to “canceled”. And then your onUpdate trigger will look something like this:
export const onUpdateEventCancel functions.firestore.document(“/events/{eventId}”).onUpdate(async change => {
const eventBeforeUpdate = change.before.data();
const eventAfterUpdate = change.after.data();
// check if event is canceled
if(eventBeforeUpdate.expirationStatus !== eventAfterUpdate.eventAfterUpdate && eventAfterUpdate.expirationStatus === ‘canceled’) {
const project = json.parse(process.env.FIREBASE_CONFIG!).projectId;
const location = ‘us-central1’;
const queue = ‘my-scheduler’;
const taskClient = new CloudTasksClient();
// task details
const eventId = eventAfterUpdate.eventId;
const expireSeconds = eventAfterUpdate.expireSeconds;
// taskName
const taskName = ‘projects/${project}/locations/${location}/queues/${queue}/tasks/task-${eventId}-${expireSeconds}’;
await taskClient.deleteTask({ name: taskName });
}
});Note: If you want to update the task in the queue, there’s no built-in method for that, but instead you need to delete and recreate the task in the queue.
Also you can store the taskName in the firestore and can directly cancel the task using that name. But for that you need a little more code in the onCreate trigger.
In the end of the onCreate trigger do something like this:
. . .
const [ response ] = await tasksClient.createTask({ parent: queuePath, task });
const taskName = { response.name };
await admin.firestore().collection(“events”).doc(eventId).update({ taskName: taskName });
. . .For that make sure you also have a taskName field in your event document.
Next your onUpdateEventCancel trigger will look something like this:
export const onUpdateEventCancel functions.firestore.document(“/events/{eventId}”).onUpdate(async change => {
const eventBeforeUpdate = change.before.data();
const eventAfterUpdate = change.after.data();
// check if event is canceled
if(eventBeforeUpdate.expirationStatus !== eventAfterUpdate.eventAfterUpdate && eventAfterUpdate.expirationStatus === ‘canceled’) {
const taskClient = new CloudTasksClient();
await taskClient.deleteTask({ name: eventAfterUpdate.taskName });
}
});Great Job: You have successfully completed TTL stuff and canceling tasks — I know this time it was a bit terrible, taking some rest may relax your neurons 😀.
Wait a second, what about security?
You might be rubbing your chin, thinking: “This is awesome, but won’t it allow just about anyone to erase my documents if they figure out my URL and payload pattern for the callback function?” The answer, surprisingly, is yes. As is, the Firebase CLI launches HTTP functions that anyone with internet access can invoke. But fear not, this potential issue has a solution.
And that is, we need to make some changes in both the task configuration and the function to use a service account to authenticate the function invoker. Here will require some advanced configuration that will not be visible on the Firebase console.
First create a service account by going to IAM & Admin > Service Accounts. On the top menu you will see CREATE SERVICE ACCOUNT. Name your service account and go for CREATE AND CONTINUE. In the second step give the following roles to the service accounts (Cloud Function Invoker, Cloud Tasks Enqueuer, and Service Account Token Creator) and you are done with the service account.
Next, on IAM & Admin come to the first tab named IAM. Grant access and in the first field “New principals” add your service account there and grant the above roles to it and save it.
You can read about IAM for Cloud Functions.
Modify onCreateEvent trigger for security
After having your service account with certain roles it’s time to dive into code again and protect our HTTP request from outside access by authenticating the function invoker with the service account.
export const onCreateEvent = functions.firestore.document("/events/{eventId}").onCreate(async snapshot => {
...
// include this in the onCreateTrigger
const serviceAccountEmail = “SERVICE_ACCOUNT_EMAIL_HERE”;
// configure the task
const task {
...
headers: { ‘Content-Type’: ‘application/json’ }
}
oidcToken: {
serviceAccountEmail,
audience: “https://${location}-${project}.cloudfunctions.net/schedulerCallback"
}
...
}
...
});In the code above, we modify the onCreate event trigger by adding one field of serviceAccountEmail. And we configure the task object by adding one more field oidcToken and we pass the serviceAccountEmail and audience the same http trigger url which leads to the schedulerCallback. Other code will remain the same.
Let’s break down the above code and let’s see what it does.
We will use serviceAccountEmail to authenticate the function invoker with that serviceAccountEmail. And the audience is specified the same as the service URL (excluding query params) and this field is used to specify the intended recipient of the JWT, and this is used as a form of access control. Using the service’s URL as an audience helps to ensure that the token is used for the right service, and prevents the token from being misused to access other services.
After you have set up the service account and the configured task with the oidcToken, every time the onCreateEvent trigger fires and a task is created, Google Cloud Tasks receives the instructions on who should make the HTTP request and where it should be sent.
When the scheduled time arrives and the task is about to be dispatched, the service account specified serviceAccountEmail field is used to generate an OpenID Connect (OIDC) token. This token claims about the identity of the service account, including the audience claim that matches your Cloud Function’s URL.
The OIDC token is sent as part of the HTTP request’s Authorization header when the task is fully dispatched. When your Cloud Function receives the request, it can validate the token to ensure that the request is authenticated and authorized. It verifies the audience claim matches its own URL, confirming that the token was intended for this service.
Modify schedulerCallback for security
First of all, run this command to install the google-auth-library. We will use it to verify ID token.
npm install google-auth-libraryNext go to the index.ts and import this:
import { OAuth2Client } from “google-auth-library”;Then, create an asynchronous function that will verify the ID token.
async function verifyToken(token: string, location: any, project: any) {
const client = new OAuth2Client();
const ticket = await client.verifyIdToken({
idToken: token,
audience: “https://${location}-${project}.cloudfunctions.net/schedulerCallback”,
});
return ticket.getPayload();
}Note: Make sure you pass the audience the same as you passed previously in the oidcToken within the task object.
Now, let’s go for modifying the schedulerCallback so that every time a call is made to schedulerCallback function it will verify an id token, if it was from the accurate service account it will verify it and will allow the request to further execute and delete the document, but if it was with wrong service account and don’t have a token of authorization token in the headers it will send a response of “unauthorized token”.
The schedulerCallback function will become:
export const schedulerCallback = functions.https.onRequest(async (req, res) => {
// add the projectID and location
const project = json.parse(process.env.FIREBASE_CONFIG!).projectId;
const location = ‘us-central1’;
const eventId = req.query.eventId as string;
// access the headers > authorization of the request
const authorizationHeader = req.headers.authorization;
// check if authorization header is not null
if(!authorizationHeader) {
res.status(401).send(“unauthorized token”);
return;
}
// if authorizationHeader is not null access the token
const token = authorizationHeader.split(‘ ’)[1];
// verify ID token
try {
await verifyIdToken(token, location, project);
} catch (error) {
console.log(error);
res.status(401).send(“Unauthorized token”);
return;
}
// delete document
try {
await admin.firestore().collection(“events”).doc(eventId).delete();
} catch(e) {
console.log(e);
res.status(500).send(e);
}
});Phew! That was quite the security sprint, wasn’t it? But just like an arduous marathon, the euphoria of crossing the finish line makes it all worthwhile. Now, we have TTL documents with secure deletion, and good night sleep free from security worries.
Before we part ways, there’s one more thing to remember. Security is a never ending journey and must be a part of your application’s DNA. Make sure you frequently revisit, review, and revise your security strategies. You are the guardian of the Firestore kingdom. WIth knowledge and foresight, you can protect it from any potential invasions.
In the world of code, every line is a new adventure. Today, we’ve journeyed through TTL documents, cancellations, and security issues. And in the 2nd part of this article we will thoroughly go through the event scheduling system using Cloud Tasks so never miss out.
Let’s connect:
LinkedIn: https://www.linkedin.com/in/muhammad-adnan-23bb8821b/
GitHub: https://github.com/AdnanKhan45
Instagram: https://www.instagram.com/dev.adnankhan/
