Using Cloud Scheduler and Cloud Functions to Deploy a Periodic Compute Engine VM Worker.
In this article we use cloud scheduler to periodically startup and tear down a compute engine (GCE) worker. Cloud Scheduler will periodically poll a cloud function that will instantiate a worker compute engine VM, that will do some work until the scheduler tells the cloud function to shut it down. For our example we shall have the cloud scheduler run every 10 minutes, polling that cloud function to perform its task.
Cloud Scheduler: Google Cloud Scheduler is a cron based solution for engineers looking for a way to set up some form of cron functionality with any GCP service. We shall manually create a cron that hits a cloud functions HTTP endpoint that triggers the entire build process.
Cloud Functions: Majority of logic for creating the GCE instance will be held within Cloud Function. We shall use the Golang GCP API to perform these calls; but do not worry if you are unfamiliar with Go, I shall heavily comment my code. Which I supply at the end of the article.
Compute Engine Instance: Cloud Functions will create a GCE instance that will perform some work. In our simple example, we shall attach startup and shutdown scripts to the VM, so that whenever the VM is started or stopped, it will write to a
work.txt file in the root folder. This will let us know the VM actually started and did some work. In your case it could be running some analytics, or whatever batch job you want done. Our instance will be preemptible to save costs.
1. Create a Service Account
We need a service account (SA) that cloud functions uses to act on behalf of us (the user) to manage the creation, starting and stopping of the VM. Without it, Cloud Functions wont be authorized to do so. In my example I call the SA
compute-admin-sa, I give it a description to better identify its role (Figure 1).
1.2 Apply Service Account Permission.
In this step we define permissions our service account will have. I give it Compute Admin role which grants it full control over Compute Engine. In your case you can be more granular with the scope. After this step, we create the cloud function.
2. Creating the Cloud Function
Navigate to the Cloud Functions Page. Click Create function.
Create a function and let’s give it a descriptive name.
worker-instance-cloud-function. Let’s ensure the Trigger is HTTP as we will call this HTTP URL using Cloud Scheduler.
Next copy the URL provided. This will be how Cloud Scheduler calls this function.
Next we select the Runtime. I amusing Go for my functions logic, so Go 1.11 is appropriate.
Next we select the function to execute. In my case its called
DeployInstance (See Code at Bottom of Page)
Let’s select a region, typically we should select a region that’s close to our Cloud Scheduler and Compute Engine Instance, in our case us-central1 works.
Let’s then select the Service Account. We use the one we created earlier.
In the Environment Variables section, my code uses 4 core variables.
INSTANCE_NAME fill those in with the variables you want. Make sure they are correct.
Finally click Create
Once deployed, you should have a similar looking UI as shown in Figure 5 below
Creating the Cloud Scheduler
It’s time to create our scheduler (Figure 6).
Let’s provide a unique descriptive name as well as a cogent description. In my case I name it
Next let’s set the scheduler to every 10 minutes. Cloud Scheduler uses the familiar Linux Cron syntax. In our case it is
*/10 * * * *
Next set the timezone to one similar to your Cloud Function region for uniformity.
Next set the Target as HTTP and paste in the Cloud Function URL we got from creating our Cloud Function. Then select
GET as the HTTP method, as we just want to call the function, passing in no data.
Running the Schedule
We can run the schedule to see if it works, On the Cloud Scheduler Page, click run now (Figure 7), if it hasn’t run already (Or you can wait 10 minutes)
If the call was successful you should see the result column on the right show Success (Figure 8)
Let’s navigate to the compute engine page. You should initially see the following in Figure 9
After a few moments, we should see our VM come to existence (Figure 10)!
The startup and shutdown scripts write to a file and also showcase the time that the VM was
STOPPED. We can use this for reference.
I left the schedule to run throughout the night, and this is what we get when viewing the Cloud Function. We can see periodic spikes in utilization every 10 minutes, showing our function was being executed (See Figure 11)
Here is the output for the
And there we have it, we have pulled together a cloud scheduler, cloud functions and compute engine.
GitHub Link (It was too long to paste in this doc)