Cloud Logging events: Add rate limiting to preserve resources

Event Driven pattern is very popular, especially in the cloud ecosystem able to generate events on many sources. In addition, the serverless architecture and their elasticity is the best place to manage those unpredictable event spikes, throughput, or even missing events!

On Google Cloud, Eventarc is the preferred solution to manage events. Some are native (Cloud Storage, Firestore,…) but many come from Cloud Logging entries, especially audit logs.

And Cloud Run service is Eventarc’s best friend for managing events in a serverless way. It scales automatically from 0 to hundreds of instances.

What’s the problem with this magic Eventarc/Cloud-Run combo?

The issue

With this standard pattern, the event is caught and delivered as soon as it is created, for immediate processing.

However, processing on Cloud Run could have some restrictions, especially if it uses rate-limited APIs, or if a low (or at least controlled) number of instances can be handled in parallel (in the case of limited connections to a database, for instance).

How can we rate-limit the delivery of events to Cloud Run?

The Cloud Task solution

When you want to perform rate limiting, Cloud Task is the ideal solution. You can stack the tasks in a queue and control the rate delivery on the queue.

The parameters max-concurrent-dispatch (number of parallel delivery) and max-dispatches-per-second (number of delivery per second) help to achieve this.

Therefore, the solution is to put Cloud Task between the event source (Cloud Logging for instance) and the event consumer (Cloud Run for instance).

However, Cloud Task is not a predefined endpoint to Eventarc. So how can we implement this architecture?

The multiple flavors of Google Cloud

As often, there are multiple ways to achieve implementation on Google Cloud. For this architecture, I will explore 2 ways, still serverless, and I will discuss the pros and cons.

The first option is to continue to use Eventarc to catch the events. However, and intermediary Cloud Run service needs to be deployed to receive the events and create the tasks in Cloud Task

The second option is to abandon Eventarc and catch directly the event from Cloud Logging (through a sink to PubSub) and then create tasks directly from a PubSub push subscription.

Here, a schema to summarize the 2 options:

Details of the implementation and configuration, if you want to reproduce it or reuse it, are in this Github repository.

The Eventarc option

This option continues to use Eventarc and its capacity to handle a large variety of events.

The downside is its limited supported delivery endpoints:

• Cloud Run and Cloud Functions backend on Cloud Run (Cloud Functions 2nd gen)
• Google Kubernetes Engine (GKE
• Internal HTTP endpoints in a Virtual Private Cloud (VPC) network
• Workflows

Therefore, a new Cloud Run service must be deployed only to get the events and stack them on Cloud Task Queue.

The service is simple, but requires code, deployment, (minimal) maintenance. And you will pay for CPU and memory to do this simple conversion.

However, the good side of this custom forwarding is the capacity to route, convert and enrich the events in that step before creating the task in Cloud Task. In particular, you can change the task destination URL or task delivery time according to the event type or header.

The Cloud Logging sink option

This solution takes us back to the past when Eventarc did not exist. At this time, the equivalent solution was to create a Cloud Logging filter and to sink (publish) each matching log entries to a PubSub topic.

However, having events in PubSub doesn’t address the need for task creation in a queue. For this, we leverage 2 new features of Cloud Tasks:

• Task buffer offers the ability to create a simple task, with minimal (no) customization.
• Task queue URL override allows you to disregard the task destination URL and to always override it with a predefined URL

These 2 features enable a PubSub push subscription to post a simple event to Cloud Task Buffer endpoint. This creates a non-customized task, and the task queue configuration determines the destination URL.

For this option, I’ve also incorporated PubSub’s “no wrapped messages” feature. By default, PubSub wraps the messages in its standard format, requiring the consumer to understand this format to unwrap and read the base64-encoded payload. The no-wrapper option allows you to push the raw message without the PubSub formatting.

Which one to use?

The best approach depends on your specific circumstances. Here’s a summary of the pros and cons of each option:

In my opinion:

• The Eventarc option is easier to understand and more customizable (with less code). However, it can be more expensive if you have a high volume of events.
• The Cloud Logging option is elegant due to its minimal code requirements (only cloud services configuration) but is less versatile and better suited for advanced/craftsman users.

Don’t hesitate to share your opinion to improve this comparison!

Just for the fun

A closer look at Eventarc’s behavior reveals that it automatically creates topics and subscriptions in your project, with the eventarc prefix in the resource name followed by the name of Eventarc resource.

Being aware of that, you can be creative and imagine a hack!
Of course, this option is not recommended in production because it can break without warning

The idea here is to let Eventarc configure event handling for us (Cloud Logging listener, PubSub topic and subscription).
Then, we will update the newly created PubSub subscription to route the events to the Cloud Task URL override queue.

Here the new architecture schema (hack path is in red)

Like this, you avoid the intermediary Cloud Run service, while using the power of Eventarc event management (easy configuration, wide range of event type,…).
Because it’s a hack, you can expect side effects when you update your Eventarc configuration as well as the possibility of this option disappearing in a future Eventarc release.

But it’s just fun to make something working while it wasn’t designed for!

Don’t fear the event driven

Managed services, serverless services, event driven architectures, automatic scaling might seem intimidating and make you feel like you have less control and fewer customization options. However, as demonstrated here, you have elegant solutions to address these challenges in either case.

Choose the option that best fits your needs, be creative in your implementation, and take your event-driven architecture to the next level!