Architecture diagram for Article

Using Cloud Scheduler and Cloud Functions to Deploy a Periodic Compute Engine VM Worker.

In this article we use cloud scheduler to periodically startup and tear down a compute engine (GCE) worker. Cloud Scheduler will periodically poll a cloud function that will instantiate a worker compute engine VM, that will do some work until the scheduler tells the cloud function to shut it down. For our example we shall have the cloud scheduler run every 10 minutes, polling that cloud function to perform its task.

Architecture

Cloud Scheduler: Google Cloud Scheduler is a cron based solution for engineers looking for a way to set up some form of cron functionality with any GCP service. We shall manually create a cron that hits a cloud functions HTTP endpoint that triggers the entire build process.
Cloud Functions: Majority of logic for creating the GCE instance will be held within Cloud Function. We shall use the Golang GCP API to perform these calls; but do not worry if you are unfamiliar with Go, I shall heavily comment my code. Which I supply at the end of the article.
Compute Engine Instance: Cloud Functions will create a GCE instance that will perform some work. In our simple example, we shall attach startup and shutdown scripts to the VM, so that whenever the VM is started or stopped, it will write to a work.txt file in the root folder. This will let us know the VM actually started and did some work. In your case it could be running some analytics, or whatever batch job you want done. Our instance will be preemptible to save costs.

Let’s begin.

1. Create a Service Account

We need a service account (SA) that cloud functions uses to act on behalf of us (the user) to manage the creation, starting and stopping of the VM. Without it, Cloud Functions wont be authorized to do so. In my example I call the SAcompute-admin-sa, I give it a description to better identify its role (Figure 1).

Figure 1: Creating the Service Account

1.2 Apply Service Account Permission.

In this step we define permissions our service account will have. I give it Compute Admin role which grants it full control over Compute Engine. In your case you can be more granular with the scope. After this step, we create the cloud function.

Figure 2: Applying Roles to Service Account

2. Creating the Cloud Function

Navigate to the Cloud Functions Page. Click Create function.

Figure 3: Creating Cloud Function (Name, Description and Trigger)

Create a function and let’s give it a descriptive name. worker-instance-cloud-function. Let’s ensure the Trigger is HTTP as we will call this HTTP URL using Cloud Scheduler.
Next copy the URL provided. This will be how Cloud Scheduler calls this function.

Next we select the Runtime. I amusing Go for my functions logic, so Go 1.11 is appropriate.

Figure 4: Creating Cloud Function (Regions and Environment Variables)

Next we select the function to execute. In my case its called DeployInstance (See Code at Bottom of Page)

Let’s select a region, typically we should select a region that’s close to our Cloud Scheduler and Compute Engine Instance, in our case us-central1 works.

Let’s then select the Service Account. We use the one we created earlier.

In the Environment Variables section, my code uses 4 core variables. PROJECT_ID, REGION, ZONE, INSTANCE_NAME fill those in with the variables you want. Make sure they are correct.

Finally click Create

Once deployed, you should have a similar looking UI as shown in Figure 5 below

Figure 5: Cloud Scheduler after you created the schedule

Creating the Cloud Scheduler

It’s time to create our scheduler (Figure 6).

Figure 6: Creating the Cloud Schedule

Let’s provide a unique descriptive name as well as a cogent description. In my case I name it worker-instance-batchwork-scheduler .

Next let’s set the scheduler to every 10 minutes. Cloud Scheduler uses the familiar Linux Cron syntax. In our case it is */10 * * * *

Next set the timezone to one similar to your Cloud Function region for uniformity.

Next set the Target as HTTP and paste in the Cloud Function URL we got from creating our Cloud Function. Then select GET as the HTTP method, as we just want to call the function, passing in no data.

Running the Schedule

We can run the schedule to see if it works, On the Cloud Scheduler Page, click run now (Figure 7), if it hasn’t run already (Or you can wait 10 minutes)

Figure 7: Cloud Scheduler immediately you created the schedule

If the call was successful you should see the result column on the right show Success (Figure 8)

Figure 8: A successful call to Cloud Functions

Let’s navigate to the compute engine page. You should initially see the following in Figure 9

Figure 9: There are no compute engine VMs initially

After a few moments, we should see our VM come to existence (Figure 10)!

Figure 10: After a few moments, the worker-instance-01 is creating

The startup and shutdown scripts write to a file and also showcase the time that the VM was STARTED or STOPPED. We can use this for reference.

I left the schedule to run throughout the night, and this is what we get when viewing the Cloud Function. We can see periodic spikes in utilization every 10 minutes, showing our function was being executed (See Figure 11)

Figure 11: Cloud Function periodic utilization

Here is the output for the work.txt.

Figure 12: The work.txt file and all the logs from STARTING to FINISHING as well as time

Conclusion

And there we have it, we have pulled together a cloud scheduler, cloud functions and compute engine.

Code

GitHub Link (It was too long to paste in this doc)

Additional Links

Startup Scripts

Preemptible VMs