Task Scheduling made easy by Google Cloud Scheduler — A managed cron service

Although late in the game but, Google Cloud is getting a managed cron service (though it had manual cron service for app engine) called Cloud Scheduler, ought to provide all the functionality of the kind of standard command-line cron service you probably love to hate, but with the reliability and ease of use of running as a managed service in cloud.

Scheduling Services by major Public Cloud Providers

AWS rolled out it’s Batch service (dedicated scheduler service) in late 2016. Microsoft’s Azure scheduler service became generally available in late 2015. Now, Google Cloud Scheduler is another step in the positive direction by Google to attract enterprise customers which run large quantity of regular batch jobs such as database updates, backups and report generation.

What is Google Cloud Scheduler?

Cloud Scheduler is an enterprise grade job scheduler that will help you automate your jobs across various Google Services in a standard fashion and comes with retry mechanisms that you can configure. It can also trigger your jobs in a variety of ways and currently supports invoking a HTTP endpoint, send a message to Cloud Pub/Sub and trigger an internal App Engine URL. This service is in beta at this moment.

How did we schedule jobs in GCP thus far?

There were a couple of alternates available in GCP so far to schedule tasks.

1. We have App engine cron service which can be used to schedule a task running on app engine or to raise a pub/sub message which may further trigger a task running on compute engine VM.
2. We can also have cron setup (using traditional command-line service on unix systems) on a compute engine VM machine as long as we have application running on one VM only. In case of distributed systems, this approach fails.

All these approaches have some logistical overhead. For one, developers must manage the underlying infrastructure connected with cron and restart jobs manually if one doesn’t complete properly. And checking to see if a cron job has run successfully involves some manual labor. Google Cloud Scheduler claims to mask all this complexity.

How to use Cloud Scheduler?

From the GCP console main menu, visit the Cloud Scheduler page:

From Google Cloud Console main menu

Click on Cloud Scheduler option in Tools sub-menu. Since we don’t have any job as yet, we will see a dialog as shown below:

Click on Create Job and fill out the form that appears, see here:

Here are a few details about a few key fields shown above:

• Name: Unique name across all jobs in the project.
• Frequency: Provide the cron expression using unix-cron format. I kept it as * * * * * since I want to trigger this job every minute.
• Target: the Cloud Scheduler currently supports App Engine, HTTP and Pub/Sub. We will select Pub/Sub and once this job is triggered (as per schedule), a message will land on Pub/Sub topic, which subscriptions can listen to and start the desired batch job.
• Topic: pass the topic name. Please do create a topic with same name and a subscription for it.
• Payload: Pass a test message.

Now click the Create button. This will create the job and job listing would be updated. Now, you can run this job from GCP console (as shown below), or from Cloud Scheduler API, or from CLI (such as Cloud Shell)

Jobs listing in Google Cloud Scheduler

You can even view logs of execution from console in stack driver.

Now, let’s verify the results.

• Go to cloud shell and execute the following command:

gcloud pubsub subscriptions pull cron-sub --limit 5

Assuming your subscription name is cron-sub. You will see some test message results.

Use Cases Covered

Now that the scheduling a task/job has become seamless on GCP, it will help automate various enterprise use cases than ever before:

1. Say; you have a moderate task (e.g. trigger database backup, trigger CI/CD pipelines, sending emails, trigger push notifications etc) to be performed on a scheduled basis:

a) you may write a cloud functions for the task functionality.

b) Use ‘HTTP’ as target while creating a job in Cloud Scheduler and pass cloud function URL to be invoked.

You could even have complex scheduling like run the Cloud Function every Monday, Wednesday and Friday at 06:00 AM.

2. Say; you have a heavy duty process (e.g. generate database reports etc) which needs to run on a Compute Engine VM on a scheduled basis:

a) Write/develop your process in any language of you choice and deploy on a VM.

b) Create a job on Cloud Scheduler having target as Pub/Sub, which on scheduled time will generate a message on pub/sub topic and it’s subscription can kick start the process on VM.

Pricing

Google provides developers with a free quota of three (3) jobs per month, per account. Additional jobs cost $0.10 per month. Note that the free tier is measured at the account level not the project level. For example, if your account has 5 projects with 2 jobs each, you will have 3 free jobs and 7 paid jobs. Follow this official pricing page for any changes in pricing in the future.

Clean up

Don’t forget to clean up scheduler jobs, pub/sub topic and subscription etc.

Please feel free to share your feedback or if you want more articles around this topic!