Introducing Eventarc in Pic-a-Daily Serverless Workshop

Pic-a-Daily Serverless Workshop

As you might know, Guillaume Larforge and I have a Pic-a-Daily Serverless Workshop. In this workshop, we build a picture sharing application using Google Cloud serverless technologies such as Cloud Functions, App Engine, Cloud Run and more.

We recently added a new service to the workshop. In this blog post, I want to talk about the new service. I also want to talk about Eventarc and how it helped us to get events to the new service.

Previous architecture

This was the previous architecture of Pic-a-Daily:

To recap:

  1. A user uploads a picture to the pictures bucket via a frontend.
  2. Image analysis Cloud Function receives the upload event, extracts the metadata of the image and saves to Firestore.
  3. Thumbnail service receives the upload event via Pub/Sub, resizes the image and saves to the thumbnails bucket.
  4. Collage service creates a collage of thumbnail pictures.

Image garbage collector service

One problem of the previous architecture was around image deletions. When an image was deleted from the pictures bucket, the thumbnail of the image in thumbnails bucket and the metadata of the image in Firestore lingered around.

To fix this problem, we created a new Cloud Run service, image garbage collector, to clean up after an image is deleted.

To get Cloud Storage events to image garbage collector, we could have used Pub/Sub pushed messages, as we did in thumbnails service. However, we decided to use Eventarc instead.

What is Eventarc?

We announced Eventarc back in October as a new eventing functionality that allows you to send events to Cloud Run from more than 60 Google Cloud sources. This is done by reading AuditLogs from various sources and sending them to Cloud Run services as CloudEvents. There is also the option of reading events from Pub/Sub topics for custom applications.

We decided to try Eventarc as a new way to get events. We also wanted to avoid all the setup we had to do in thumbnail service to get Cloud Storage events to to Cloud Run via Pub/Sub. See details.

Current architecture

The current architecture now looks like this with Eventarc and image garbage collector:

Image garbage collector code

You can check out the full code for image garbage collector here. I’ll highlight some parts of the code.

Image garbage collector receives AuditLogs wrapped into CloudEvents from Eventarc. It first reads the CloudEvent using the CloudEvents SDK:

const { HTTP } = require("cloudevents");
...
const cloudEvent = HTTP.toEvent({ headers: req.headers, body: req.body });

The data field of the CloudEvent has the actual AuditLog. There's Google Events library to parse the data into an AuditLog:

const {toLogEntryData} = require('@google/events/cloud/audit/v1/LogEntryData');...const logEntryData = toLogEntryData(cloudEvent.data);

The rest of the code is to extract bucket and object name from the event and then delete it from thumbnails bucket and Firestore collection:

const tokens = logEntryData.protoPayload.resourceName.split('/');
const bucket = tokens[3];
const objectName = tokens[5];
// Delete from thumbnails
try {
await storage.bucket(bucketThumbnails).file(objectName).delete();
console.log(`Deleted '${objectName}' from bucket '${bucketThumbnails}'.`);
}
catch(err) {
console.log(`Failed to delete '${objectName}' from bucket '${bucketThumbnails}': ${err}.`);
}
// Delete from Firestore
try {
const pictureStore = new Firestore().collection('pictures');
const docRef = pictureStore.doc(objectName);
await docRef.delete();
console.log(`Deleted '${objectName}' from Firestore collection 'pictures'`);
}
catch(err) {
console.log(`Failed to delete '${objectName}' from Firestore: ${err}.`);
}

Create a Trigger

Lastly, we need to connect events from Cloud Storage to the new service via Eventarc. This is done with a trigger:

gcloud beta eventarc triggers create trigger-${SERVICE_NAME} \
--destination-run-service=${SERVICE_NAME} \
--destination-run-region=europe-west1 \
--matching-criteria="type=google.cloud.audit.log.v1.written" \
--matching-criteria="serviceName=storage.googleapis.com" \
--matching-criteria="methodName=storage.objects.delete" \
--service-account=${PROJECT_NUMBER}-compute@developer.gserviceaccount.com

Notice that we’re matching for AuditLogs for Cloud Storage and specifically looking for storage.objects.delete events.

If you want to try out our workshop and the new lab, here are some links:

As always, feel free to reach out to me on Twitter @meteatamel for any questions/feedback.

Originally published at https://atamel.dev.

--

--

--

A collection of technical articles and blogs published or curated by Google Cloud Developer Advocates. The views expressed are those of the authors and don't necessarily reflect those of Google.

Recommended from Medium

How Weak Analogies About Software Can Lead Us Astray

My experience with the Google foobar challenge and tips on what to do if you get it

Migrating to the Cloud Without Screwing It Up

Managed Apache Airflow on AWS — New AWS Service For Data Pipelines

RxJava 2 chaining and conventional code side by side comparison

Circuit Breaker Pattern (Design Patterns for Microservices)

Scrum and Kanban: What’s the difference?

Node.js on Heroku: A More Complete Tutorial: Part 2

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Mete Atamel

Mete Atamel

Developer Advocate at Google

More from Medium

Automatic Release Propagation for Canary releases with Cloud Run

Cloud Bigtable: What is it? For what is it suitable? And what does it cost?

Use Asgardeo as IDP in your Choreo application

LFaaS in Action: Deploy Your First Data Process in 5 minutes w/JavaScript on LOC Studio