Create Cloud Spanner Scheduled Backups

Hengfeng Li
May 29, 2020 · 5 min read

This tutorial shows how to use the Scheduled Backups tool to configure Cloud Scheduler for creating Cloud Spanner backups on a regular basis, e.g. daily or weekly.

We will use the following GCP services:

  • Cloud Scheduler: trigger tasks with a cron-based schedule.
  • Cloud Pub/Sub: a message queue from Cloud Scheduler to Cloud Functions.
  • Cloud Functions: start an operation for creating a Cloud Spanner backup.
  • Cloud Logging: create logs-based metrics.
  • Cloud Monitoring: create alerts based on conditions of logs-based metrics.

The architecture is as follows:

We will need to do the following steps:

  • Create a pub/sub topic
  • Deploy a function to Cloud Functions
  • Deploy scheduled jobs using Cloud Scheduler
  • Set up email notifications of failures

Prerequisite

Before proceeding with the tutorial, we need to make sure that

  • a database in Cloud Spanner exists (you can try this sampledb to create a sample database), and
  • the Google SDK is installed.

Create a pub/sub topic

We create a pub/sub topic cloud-spanner-scheduled-backups:

Then, go to Pub/Sub UI and you can see the newly created topic:

Deploy a function to Cloud Functions

We have a function SpannerCreateBackup in backups.go. The SpannerCreateBackup function is called whenever a pub/sub message comes in. A pub/sub message contains the database name, the backup ID, and the expiration duration (at least 6 hours). After parsing the message, it starts a CreateBackup operation in Cloud Spanner.

We can deploy the function by running the following command:

Go to the Cloud Functions UI to find the function. The function subscribes to the pub/sub topic cloud-spanner-scheduled-backups.

Now, you can test it via gcloud:

This will directly trigger an execution of the function leading to the creation of a backup.

Deploy scheduled jobs using Cloud Scheduler

Note: To use Cloud Scheduler, we must create an App Engine app.

There is a schedule template file schedule-template.yaml under Scheduled Backups. We need to make a copy and rename it to schedule.config.yaml. Then, we can replace PROJECT_ID, INSTANCE_ID, and DATABASE_ID with our configuration:

Also, this configuration file can be used to adjust the schedule. At any point it is possible to update the fields, such as schedule, expire, and time zone, and then redeploy the job to Cloud Scheduler. crontab guru is a handy website to generate a Cron schedule.

Once we have updated the configuration, we can deploy it to Cloud Scheduler:

Then, we can find the job in the Cloud Scheduler UI:

Set up email notifications of failures

To get email notifications, we need to do three steps:

  • Set up an email notification channel
  • Add logs-based metrics
  • Create alerting policies

First, we can follow this guide to add our email address as a notification channel in Cloud Monitoring. This needs to be done before we can use them in alerting policies.

Next, we can add logs-based metrics in Cloud Logging by using the Deployment Manager. These custom metrics are based on filtering logs generated by Cloud Functions, Cloud Scheduler, and Cloud Spanner. We can find definitions of our custom metrics in metrics.yaml.

We can run the following command to create our custom metrics:

Finally, we need to create alerting policies which define when to trigger an alert. The easiest way is to go to Logs-based Metrics under Cloud Logging and for each user-defined metric, there is an option “Create alert from metric”.

From there, we can choose sum as Aggregator for the target metric, and select the option most recent value in the condition, which will trigger an alert when any time series violates that the value is above 0.

After defining the condition, we’ll be brought to a site to define an alerting policy. We need to give a name to this alerting policy and set up the notification channel. Whenever the condition breaks and an alert is created, we will receive email notifications.

Summary

Scheduled Backups shows you how to use Cloud Scheduler and Cloud Functions to configure a schedule for creating Cloud Spanner backups. Also, it demonstrates how to create logs-based metrics and alerting policies. They are used for sending alerts when the predefined condition breaks so that we can get prompt notifications. The whole project provides a solution for creating Cloud Spanner backups with a defined schedule.

Google Cloud community articles and blogs

Google Cloud - Community
Hengfeng Li

Written by

PhD. Software engineer at Google. Living in Australia.

Google Cloud - Community

A collection of technical articles and blogs published or curated by Google Cloud Developer Advocates. The views expressed are those of the authors and don't necessarily reflect those of Google.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store