Recursive calling in HTTP based trigger function in GCP using .Net

Source: Google Cloud

In the realm of cloud computing, where flexibility, scalability, and efficiency reign supreme, Google Cloud Platform (GCP) shines as a leader. One of its standout services is GCP Cloud Functions, which has transformed how developers tackle serverless computing. In this deep dive, we delve into GCP Cloud Functions, exploring their architecture, advantages, practical applications, and how they empower developers to create robust, event-driven applications.

Introduction

Cloud Functions are like code pieces that react to events. They use event details to give results for users or services. These events can be generated by HTTP requests, File transfers, or Pub/Sub events. There are two types of Cloud Functions:

• HTTP event-based, which triggers through HTTP requests.
• Service event-based, which starts with specific events.

HTTP Cloud Functions are easy to configure and activate. Recursive calls are challenging. The stateless design causes timeouts post-execution.

HTTP Triggered Function

Storage Driven Event Function

Problem Statement

We need a cloud function to run several BigQuery stored procedures. Each procedure can take up to 45 minutes to complete. Creating separate functions incurs high container and operational expenses.

Objective of the Design

• Our goal is to design a single function.
• This function should:

1. Generate queries for specific stored procedures.
2. Manage recursive calls efficiently.
3. Minimise operational expenses.

Role of .Net in GCP

Google Cloud Platform (GCP) supports various programming languages, including .Net. Documentation for some languages, like .Net, is limited. GCP is constantly evolving. GCP has various APIs for .Net developers, but there is little instruction on how to use them. In this article and future ones, I’ll share my experience and thoughts on using .Net in GCP to close this gap. Let’s start by figuring out how to make, use, and run a cloud function in .Net. Feel free to ask any questions for further clarification. Thank you!

Prerequisite

Before we delve into the example, I recommend exploring links section to gain a basic understanding of the following Google Cloud services:

• BigQuery
• Cloud Scheduler
• Identity and Access Management
• Cloud Function
• Cloud Run, and more.

Design Diagram and Implementation

Let’s explore the design diagram below, illustrating the flow of the cloud function we’ll be building. We’ll begin with an HTTP-triggered cloud function called “BigQuery Trigger,” responsible for executing a BigQuery command. Using .Net code, it will create a scheduler and trigger it to make a recursive call to itself. Once the call is successful, the function will delete the scheduler using .Net code. For error handling, you can implement your own mechanisms, which we won’t cover in this article.

Architectural Design Diagram

Let’s follow a step-by-step tutorial to create and deploy our function on Google Cloud. Simply replace the Function.CS file with the provided code below.

HTTP Tutorial (2nd gen) | Cloud Functions Documentation | Google Cloud

Function.cs

using Google.Cloud.Functions.Framework;
using Microsoft.AspNetCore.Http;
using System.Threading.Tasks;
using Google.Cloud.BigQuery.V2;
using Google.Apis.Auth.OAuth2;
using System;
using System.Collections.Generic;
using System.Text;
using Google.Apis.CloudScheduler.v1;
using Google.Apis.CloudScheduler.v1.Data;

namespace HelloWorld;

public class Function : IHttpFunction
{
    private BigQueryClient _bqClient = BigQueryClient
                                              .Create(---your-project-id---);
    
    //Step 1
    private async Task<string> GetProcedureName(int activityId)
    {
        string name = null;

        switch (activityId)
        {
            case 0:
                name = await _bqClient
                              .GetRoutineAsync(---first-routine-name---)
                              .FullyQualifiedId;
                break;

            ---Other-Cases---

            default:
                name = null;
                break;
        }

        return name;
    }

    //Step 3
    private async Task<string> GenerateToken()
    {
        string token = null;
        var oidcToken = await GoogleCredential.GetApplicationDefault()
                        .GetOidcTokenAsync(OidcTokenOptions.
                          FromTargetAudience(
                              ---url-of-your-deployed-function---
                          )
                          .WithTokenFormat(OidcTokenFormat.Standard)
                        ).ConfigureAwait(false);

        if (oidcToken != null)
        {
            token = await oidcToken.GetAccessTokenAsync().ConfigureAwait(false);
        }

        return token;
    }

    //Step 2
    private async Task RunJob(int activityId, string procedureParams = null)
    {
        var impersonatedCredential = GoogleCredential.GetApplicationDefault();

        var projectLocation = "projects/"
                    + ---your-project-id---
                    + "/locations/"
                    + --- location-of-your-function ---;

        var uri = new StringBuilder(---url-of-your-deployed-function--- + "?");
        uri.Append($"id={activityId.ToString()}&params={procedureParams}");

        CloudSchedulerService cloudScheduler = new CloudSchedulerService(
            new Google.Apis.Services.BaseClientService.Initializer()
            {
                HttpClientInitializer = impersonatedCredential,
                GZipEnabled = false
            }
        );

        IDictionary<string, string> header = new Dictionary<string, string>
        {
            { "Content-Type", "application/json" },
            { "Authorization", "Bearer " + await GenerateToken() }
        };

        HttpTarget httpTarget = new HttpTarget()
        {
            Headers = header,
            HttpMethod = "POST",
            Uri = uri.ToString(),
        };

        Job job = new Job()
        {
            Description = "",
            HttpTarget = httpTarget,
            Name = projectLocation
                    + "/jobs/"
                    + activityId,
            Schedule = "5 * * * *", //cron formate
            TimeZone = "Asia/Kuwait",

        };

        cloudScheduler.Projects.Locations.Jobs.Create(job, projectLocation).Execute();
        cloudScheduler.Projects.Locations.Jobs.Run(null, job.Name).Execute();
        cloudScheduler.Projects.Locations.Jobs.Delete(job.Name).Execute();
    }

    public async Task HandleAsync(HttpContext context)
    {
        int activityId = Convert.ToInt32(context.Request.Query["id"]);
        string procedureParams = context.Request.Query["params"];
        string procedureName = await GetProcedureName(activityId);

        if (!string.IsNullOrEmpty(procedureName))
        {
            await _bqClient.ExecuteQueryAsync($"call `{procedureName}`('{procedureParams}')", null);
            await RunJob(activityId + 1);
        }
    }


}

The code consists of three main steps:

• Getting the fully qualified name (project-name.dataset-id.procedure-name) for the provided ID. If null, the program will stop.
• Creating a scheduler, executing the scheduler job, and then deleting the scheduler.
• Generating a JWT token to run the scheduler with the appropriate scopes.

To use the code, replace the following flags with the values specific to your deployed function:

• your-project-id: The ID of the Google Cloud project you're working on.
• url-of-your-deployed-function: The URL of the function after it's deployed to GCP or your localhost URL.
• first-routine-name: The name of the routine to be called.
• location-of-your-function: The location in which the function is deployed.

Conclusion

In conclusion, the challenge of designing a cloud function to execute multiple BigQuery stored procedures sequentially has been addressed through the creation of a single, efficient solution. By consolidating the execution of procedures within a single function, the operational costs associated with multiple function invocations have been minimized. The function is capable of dynamically generating queries to call specific stored procedures, enabling flexible execution while efficiently managing recursive invocations. This design ensures the seamless execution of procedures while optimising resource utilisation and operational efficiency in the Google Cloud Platform environment.

Reference Links

• Introduction to Cloud Functions:

Cloud Functions overview | Cloud Functions Documentation | Google Cloud

• Introduction to Bigquery:

BigQuery overview | Google Cloud

• Introduction to Cloud Run:

What is Cloud Run | Cloud Run Documentation | Google Cloud

• Introduction to Cloud Scheduler:

About Cloud Scheduler | Cloud Scheduler Documentation | Google Cloud

• Introduction to Identity Access Management:

IAM overview | IAM Documentation | Google Cloud