Introducing Eventarc in Pic-a-Daily Serverless Workshop

Mete Atamel
Nov 30, 2020 · 4 min read

Pic-a-Daily Serverless Workshop

As you might know, Guillaume Larforge and I have a Pic-a-Daily Serverless Workshop. In this workshop, we build a picture sharing application using Google Cloud serverless technologies such as Cloud Functions, App Engine, Cloud Run and more.

Image for post
Image for post

We recently added a new service to the workshop. In this blog post, I want to talk about the new service. I also want to talk about Eventarc and how it helped us to get events to the new service.

Previous architecture

This was the previous architecture of Pic-a-Daily:

Image for post
Image for post

To recap:

  1. A user uploads a picture to the pictures bucket via a frontend.
  2. Image analysis Cloud Function receives the upload event, extracts the metadata of the image and saves to Firestore.
  3. Thumbnail service receives the upload event via Pub/Sub, resizes the image and saves to the thumbnails bucket.
  4. Collage service creates a collage of thumbnail pictures.

Image garbage collector service

One problem of the previous architecture was around image deletions. When an image was deleted from the pictures bucket, the thumbnail of the image in thumbnails bucket and the metadata of the image in Firestore lingered around.

To fix this problem, we created a new Cloud Run service, image garbage collector, to clean up after an image is deleted.

To get Cloud Storage events to image garbage collector, we could have used Pub/Sub pushed messages, as we did in thumbnails service. However, we decided to use Eventarc instead.

What is Eventarc?

We announced Eventarc back in October as a new eventing functionality that allows you to send events to Cloud Run from more than 60 Google Cloud sources. This is done by reading AuditLogs from various sources and sending them to Cloud Run services as CloudEvents. There is also the option of reading events from Pub/Sub topics for custom applications.

We decided to try Eventarc as a new way to get events. We also wanted to avoid all the setup we had to do in thumbnail service to get Cloud Storage events to to Cloud Run via Pub/Sub. See details.

Current architecture

The current architecture now looks like this with Eventarc and image garbage collector:

Image for post
Image for post

Image garbage collector code

You can check out the full code for image garbage collector here. I’ll highlight some parts of the code.

Image garbage collector receives AuditLogs wrapped into CloudEvents from Eventarc. It first reads the CloudEvent using the CloudEvents SDK:

const { HTTP } = require("cloudevents");
...
const cloudEvent = HTTP.toEvent({ headers: req.headers, body: req.body });

The data field of the CloudEvent has the actual AuditLog. There's Google Events library to parse the data into an AuditLog:

const {toLogEntryData} = require('@google/events/cloud/audit/v1/LogEntryData');...const logEntryData = toLogEntryData(cloudEvent.data);

The rest of the code is to extract bucket and object name from the event and then delete it from thumbnails bucket and Firestore collection:

const tokens = logEntryData.protoPayload.resourceName.split('/');
const bucket = tokens[3];
const objectName = tokens[5];
// Delete from thumbnails
try {
await storage.bucket(bucketThumbnails).file(objectName).delete();
console.log(`Deleted '${objectName}' from bucket '${bucketThumbnails}'.`);
}
catch(err) {
console.log(`Failed to delete '${objectName}' from bucket '${bucketThumbnails}': ${err}.`);
}
// Delete from Firestore
try {
const pictureStore = new Firestore().collection('pictures');
const docRef = pictureStore.doc(objectName);
await docRef.delete();
console.log(`Deleted '${objectName}' from Firestore collection 'pictures'`);
}
catch(err) {
console.log(`Failed to delete '${objectName}' from Firestore: ${err}.`);
}

Create a Trigger

Lastly, we need to connect events from Cloud Storage to the new service via Eventarc. This is done with a trigger:

gcloud beta eventarc triggers create trigger-${SERVICE_NAME} \
--destination-run-service=${SERVICE_NAME} \
--destination-run-region=europe-west1 \
--matching-criteria="type=google.cloud.audit.log.v1.written" \
--matching-criteria="serviceName=storage.googleapis.com" \
--matching-criteria="methodName=storage.objects.delete" \
--service-account=${PROJECT_NUMBER}-compute@developer.gserviceaccount.com

Notice that we’re matching for AuditLogs for Cloud Storage and specifically looking for storage.objects.delete events.

If you want to try out our workshop and the new lab, here are some links:

As always, feel free to reach out to me on Twitter @meteatamel for any questions/feedback.

Originally published at https://atamel.dev.

Google Cloud - Community

Google Cloud community articles and blogs

Mete Atamel

Written by

Developer Advocate at Google

Google Cloud - Community

A collection of technical articles and blogs published or curated by Google Cloud Developer Advocates. The views expressed are those of the authors and don't necessarily reflect those of Google.

Mete Atamel

Written by

Developer Advocate at Google

Google Cloud - Community

A collection of technical articles and blogs published or curated by Google Cloud Developer Advocates. The views expressed are those of the authors and don't necessarily reflect those of Google.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store