Google Cloud Functions Tutorial : Writing Background Functions

This is part of a Google Cloud Functions Tutorial Series. Check out the series for all the articles.

In an earlier part of the series, we covered writing Foreground Functions that were based on HTTP Triggers. In this post, we are going to look at writing background functions that can be supported by events raised by two specific event providers that are supported in Google Cloud Functions: Cloud Pub/Sub and Google Cloud Storage.

Background Google Cloud Function Event Providers

This post will not be a tutorial for Cloud Pub/Sub and Google Cloud Storage. Please refer to their respective documentation on the same.

Background Functions

First up, it is important to understand a few points about Background functions:

• Unlike HTTP Trigger or Foreground functions that you invoke directly, Background functions cannot be invoked directly.
• They will be invoked as part of event triggers that occur in supported Google Cloud Platform Event Providers.
• Two Google Cloud Platform Event Providers are supported by Background Functions : Cloud Pub/Sub and Google Cloud Storage. These Event Providers are available in GA and not in Beta. Note that over time, Cloud Functions will support more Event Providers and some of them are already being supported in Beta and/or Alpha like Cloud Firestore, BigQuery and Compute Engine.
• Cloud Pub/Sub is a global message queue that is part of the Google Cloud Platform. It works on the fundamental messaging concepts like Queues, Topics, Messages, Subscribers and Subscriptions.
• Google Cloud Storage is a distributed and global blob store. You can upload and store your files, blobs into directories or buckets.
• A Background Google Cloud Function based on Cloud Pub/Sub will get triggered when a message is published to a Pub/Sub Topic. It other words, our Background Google Cloud Function will have a subscription i.e. it subscribes to messages published on a specific Pub/Sub Topic.
• A Background Google Cloud Function based on Cloud Storage will get triggered by multiple actions that can happen in Cloud Storage Buckets. For e.g. you can listen to metadata changes, file uploads, file deletes and more. Each of these events can trigger your Background Cloud Function.

Background Cloud Function Parameters

The Background Cloud Functions will be triggered as a result of the message being published to a Pub/Sub Topic or file changes in specific Google Cloud Storage buckets.

You have a choice of using Node.js 6 or Node.js 8 runtimes with Google Cloud Functions. The sections below will indicate the key differences in terms of method signatures and other points. Note that Node.js 8 brings new features to the table like async and await and so it completely depends which runtime you want to go with. Keep in mind that at the time of writing Node.js 8 is in Beta.

Node.js 6

The function signature of Background Cloud Functions will take 2 parameters:

• event
• callback

A skeletal outline for your Background function will look like this:

exports.yourfunction = (event, callback) => {
     // Extract information from event object
  
     // Your function logic
     callback();  //Invoke the callback to indicate that your 
                  //function is complete
}

The event object will contain the following properties, which can be of interest in your function logic. This is taken from the official documentation:

The event.data property will have a specific schema depending on whether the event is from the Cloud Pub/Sub event or from Google Cloud Storage event.

• Google Cloud Pub/Sub : Cloud Storage Object
• Google Cloud Storage : PubsubMessage

Node.js 8

The function signature of Background Cloud Functions will take 2 parameters:

• data
• context
• callback

A skeletal outline for your Background function will look like this:

exports.yourfunction = (data, context, callback) => {
    // Extract information from event object
  
    // Your function logic
    callback();   //Invoke the callback to indicate that your 
                  //function is complete
}

Compared to the function signature for Node.js 6 where you had to extract both the data and context from the event object, in Node.js 8 function signature, the first parameter is the data object, the 2nd parameter is context , which is provided as a separate parameter as shown above.

We will cover more on this in the respective sections for two Event Provider triggered functions.

Google Cloud Function based on Cloud Pub/Sub

Let us take a look at writing our first Cloud Function based on the Cloud Pub/Sub Event Provider. A skeletal template for the same (taken from the default code that appears) is shown below. We are covering this for Node.js 6 version for now.

Note that the event object that we mentioned has the data property that will contain the message that was published to the Pub/Sub Topic. This message is encoded in base64 format, so you will need to decode the same in your code. Similarly, take a look at the other attributes that you might be interested in your function logic.

Let us go ahead and write a Google Cloud Function based on Pub/Sub now.

The first step is to be logged into the Cloud Console with your Cloud Project selected. Click on Cloud Functions from the main menu and click on Create Function. This will lead to the Cloud Function creation dialog as shown below:

Let us look at the form parameters:

• Name : Give it some name. For e.g. pubsubfunction1, etc.
• Memory : You can select 128MB or 256MB, since this is more of a test function.
• Trigger: Select Cloud Pub/Sub topic this time.
• In the Topic dropdown, click on new Topic. We are going to use a new topic over here. This will bring up a Topic form as shown below, where you can provide the topic name. Click on CREATE.

• Runtime : Go with Node.js 6
• Go with other defaults for Inline Source Editor, function to invoke, etc. We are going to use the standard code template pre-filled with us since our intent is to understand the mechanics of the whole process.
• Click on Create button to create the function.

On successful function creation, you will be led to the function screen as shown below:

Once you click on the function, this will lead to the Function details screen. Click on the Trigger tab as shown below. Notice the Trigger type and Pub/Sub topic name.

Click on Source tab. This will lead to the source for your Cloud Function as shown below:

index.js

exports.helloPubSub = (event, callback) => {
  const pubsubMessage = event.data;
  console.log(Buffer.from(pubsubMessage.data, ‘base64’).toString());
  callback();
};

The code is straightforward:

• Notice that our function signature takes two parameters as explained before i.e. event and callback
• The event.data property will contain the contents of the message published.
• The message data is in base64 format. So the code decodes it into plain text and uses console.log to log the message. This will be eventually visible in the Stackdriver logs.
• Finally we indicate that we are done with our function execution by invoking the callback method.

Let us test our function now. Click on the Testing tab and enter the sample data as shown below. Keep in mind that we are base64 encoding our data to simulate how the message will be published to the Pub/Sub topic.

Click on the Test the function button. This will invoke our Cloud function and you can see that in the logs below.

But keep in mind that we simply used the testing message to simulate a message that was being published to the Pub/Sub topic. How about actually using Pub/Sub in Cloud Console and publish the message to the topic from there. That should then invoke our function because it has a subscription to any message published on that topic. Let’s do that.

Click on Pub/Sub from the Cloud Console main menu. In the list of topics, you should find the topic that we created while creating the Cloud Function. You will also notice that there is already a subscription present for our Topic, which means that the messages that are published to this topic will be sent to the subscribers to that topic. And it is no surprise that the subscription is one of our Cloud Function, which will get invoked when a message is published to that topic.

Click on the ellipsis at the end and it will bring up a list of options as shown below. Select Publish message since we want to publish a message to this topic.

In the Publish message form, enter the message that you want to send and click on Publish button.

This will publish the message to the topic. This in turn should invoke our Cloud Function. Since we are logging the message that is received via the console.log statements in our Cloud Function code, we can visit the Stackdriver Logging and we find that our function did get invoked successfully.

This completes the task of writing our first Cloud Function based on Pub/Sub Event Provider.

Node.js 8

So far we used the Node.js 6 runtime. In case we wanted to use the Node.js 8 runtime, we would do the following:

1. While creating the function , simply select Node.js 8 runtime.
2. The index.js code that will be provided as a template is shown below. Note that as we explained earlier the parameters are event and context , there is no callback parameter here. Since we are only interested in extracting out the data published, we continue to the use the data attribute on the event object as shown below:

exports.helloPubSub = (data, context, callback) => {
 
 const pubsubMessage = event.data;
 console.log(Buffer.from(pubsubMessage, ‘base64’).toString());
 callback();
};

Google Cloud Function based on Cloud Storage

Let us take a look at writing our first Cloud Function based on the Cloud Storage Provider. A skeletal template for the same (taken from the default code that appears) is shown below. We are covering this for Node.js 6 version for now.

/**
 * Triggered from a change to a Cloud Storage bucket.
 *
 * @param {!Object} event Event payload and metadata.
 * @param {!Function} callback Callback function to signal completion.
 */
exports.helloGCS = (event, callback) => {
 console.log(`Processing file: ${event.data.name}`);
 callback();
};

Note that the event object that we mentioned has the data property that will contain information on the Google Cloud Storage Event. It contains information on Bucket name, Content-type and other metadata too. Similarly, take a look at the other attributes that you might be interested in your function logic.

Do note that always call the callback() function or return a Promise from your function to indicate that your function processing is complete.

Let us go ahead and write a Google Cloud Function based on Cloud Storage now.

The first step is to be logged into the Cloud Console with your Cloud Project selected. Click on Cloud Functions from the main menu and click on Create Function. This will lead to the Cloud Function creation dialog as shown below:

Let us look at the form parameters:

• Name : Give it some name. For e.g. gcs-function1, etc.
• Memory : You can select 128MB or 256MB, since this is more of a test function.
• Trigger: Select Cloud Storage bucket this time.
• In Bucket field, click on Browse and create a unique Bucket name that we will be monitoring for any file uploads, metadata changes, etc.
• In the Event Type, select the Finalize/Create event. This event is created when the file has been successfully created and written to the Google Cloud Storage Bucket that we are monitoring.
• Runtime: Select Node.js 6.
• In the Source tab, please delete the source code present and replace it with the one shown below for the index.js file.

exports.helloGCSGeneric = (event, callback) => {
const file = event.data;
const context = event.context;
console.log(` Event ${context.eventId}`);
console.log(` Event Type: ${context.eventType}`);
console.log(` Bucket: ${file.bucket}`);
console.log(` File: ${file.name}`);
console.log(` Metageneration: ${file.metageneration}`);
console.log(` Created: ${file.timeCreated}`);
console.log(` Updated: ${file.updated}`);
callback();
};

Ensure that the function name that you export is helloGCSGeneric.

• Click on Create button to create the function.

On successful function creation, you will be led to the function screen as shown below:

Click on the function to view the function details (Trigger tab) as shown below. Note the Trigger type and our Bucket name that we specified.

Let us test the function now via Google Cloud Storage service in the Cloud Console. Navigate to Cloud Storage in the Cloud Console. You should see a screen as shown below:

Click on the specific bucket that we created. This will lead you to a screen as shown below:

Go ahead and click on Upload Files button. Select any file from your local machine and upload it.

Once the file is successfully uploaded, we are tracking for all changes that are happening in the gcs-function-bucket1, so our function should get invoked and we should see the required output in the Stackdriver Logging service as shown below:

This completes our task of writing a Google Cloud Function based on Google Cloud Storage.

Node.js 8

So far we used the Node.js 6 runtime. In case we wanted to use the Node.js 8 runtime, we would do the following:

1. While creating the function , simply select Node.js 8 runtime.
2. In the index.js code that is provided for the function, use the following:

exports.helloGCSGeneric = (data, context, callback) => {
 const gcsEvent = data;
 console.log(`Processing file: ${gcsEvent.name}`);
 console.log(`Event ${context.eventId}`);
 console.log(`Event Type: ${context.eventType}`);
 console.log(`Bucket: ${gcsEvent.bucket}`);
 console.log(`Metageneration: ${gcsEvent.metageneration}`);
 console.log(`Created: ${gcsEvent.timeCreated}`);
 console.log(`Updated: ${gcsEvent.updated}`); 
 
};

A few notes on Callback

As a best practice, always remember to invoke the callback from your function. This will indicate that your function processing is complete and the service can calculate the time that it has taken your function to run. If you do not invoke the callback, your function will be considered as running till it times out, which is not something that you want to do.

There are multiple ways in you can invoke the callback. They are listed below:

• A callback() method without parameters or null value indicates success. For e.g. you can indicate success like the following style too callback(null,"Function completed successfully");
• If you wish to return a failure from your function, which will get logged in Stackdriver logging as an ERROR, you can do so by specifying an Error object as the first parameter or even a value there. For e.g. callback(100) or callback(new Error('An Error message'));

You don’t have to always specify a callback() to denote function completion. You could return a Promise too or alternately return a discrete value from your function or throw an Error object from your code. Cloud Functions will handle that as a callback for you. In this case, do remember to omit the callback parameter from the function signature.

Here is an example of throwing an Error object from your code or returning a discrete value. Both of these techniques are valid ways to indicate function completion.

exports.somefunction = (event) => {
    //Your function logic
    if (some_condition === true) {
        return "a value";
    } else {
        throw new Error('An error message');
    }
};

A final way of not using the callback is to wrap your function using the async keyword (which causes your function to implicitly return a Promise).

This completes our section on writing Background Cloud Functions.

Proceed to the next part : Monitoring Cloud Functions or go back to the Google Cloud Functions Tutorial Home Page.