TECHNICAL BLOG / GOOGLE CLOUD PLATFORM

Schedule GCP Filestore Snapshot using Cron — Part II

This blog is about scheduling the filestore snapshot using cron expression. Currently such feature is not supported in filestore out of the box.

If you have not gone through Part I which includes basics around Filestore Snapshots, check below link.

[How To] Snapshot GCP Filestore — Part I

As a workaround, let’s implement the scheduling the filestore snapshot with the combination of different GCP services.

• Cloud Scheduler
• Cloud Functions
• Filestore backups

Design Flow

How the flow would look like:

1. Cloud Scheduler service will be configured using cron expression and target Cloud Function. As the schedule triggers the Cloud Scheduler would invoke execution of mentioned target Cloud Function.
2. Cloud Function service will be configured to trigger the Filestore snapshot action using gcloud REST API calls. That will result in Filestore Snapshot.

Deployment Design Diagram

Implementation Flow

We will deploy the complete design flow into 3 steps:

1. Configure a Service Account for Cloud Scheduler and Cloud Function.

In this step we will create two different service account that can be used by both services as below:

• Cloud Scheduler can trigger Cloud Function

## Create GCP ServiceAccount 
gcloud iam service-accounts create snapshotschedule \
    --display-name="Service Account for FS Snapshot Cloud Scheduler" \
    --project=umeshkumhar-1


## List GCP ServiceAccounts
gcloud iam service-accounts list --project=umeshkumhar-1

> DISPLAY NAME                                     EMAIL                                                   DISABLED
> Service Account for FS Snapshot Cloud Scheduler  snapshotschedule@umeshkumhar-1.iam.gserviceaccount.com  False


## Find the Cloud Scheduler service agent
export PROJECT_NUMBER=`gcloud projects describe umeshkumhar-1 --format='value(projectNumber)'`
export SCHEDULER_SA=service-$PROJECT_NUMBER@gcp-sa-cloudscheduler.iam.gserviceaccount.com


## Create IAM policy binding to ServiceAccount
gcloud iam service-accounts add-iam-policy-binding snapshotschedule@umeshkumhar-1.iam.gserviceaccount.com\
    --member=serviceAccount:$SCHEDULER_SA \
    --role=roles/cloudscheduler.serviceAgent

The role roles/cloudscheduler.serviceAgent allow service agent to impersonate the client service account to invoke Cloud Function.

• Cloud Function can trigger Filestore Snapshot APIs

## Create GCP ServiceAccount
gcloud iam service-accounts create snapshotfunction \
    --display-name="Service Account for FS Snapshot Cloud Function " \
    --project=umeshkumhar-1


## List GCP ServiceAccounts
gcloud iam service-accounts list --project=umeshkumhar-1

> DISPLAY NAME                                     EMAIL                                                   DISABLED
> Service Account for FS Snapshot Cloud Function   snapshotfunction@umeshkumhar-1.iam.gserviceaccount.com  False


## Create IAM policy binding to ServiceAccount
gcloud projects add-iam-policy-binding umeshkumhar-1 \
    --member=serviceAccount:snapshotfunction@umeshkumhar-1.iam.gserviceaccount.com \
    --role=roles/file.editor

2. Configure Cloud Function with the code to invoke Filestore Snapshot rest APIs.

In this step, we will create a cloud function named fs-snapshot with python code in 3.7. Before deploying the cloud function, create python file named main.py and copy below code that read variable values and accordingly trigger the filestore snapshot.

Requirements: requirements.txt

google-auth==1.19.2
requests==2.24.0

Code Snippet: main.py

PROJECT_ID = 'umeshkumhar-1'
SOURCE_INSTANCE_ZONE = 'us-central1'
SOURCE_INSTANCE_NAME = 'test-instance'
SOURCE_FILE_SHARE_NAME = 'fileshare'

import google.auth
import google.auth.transport.requests
from google.auth.transport.requests import AuthorizedSession
import time
import requests
import json

credentials, project = google.auth.default()
request = google.auth.transport.requests.Request()
credentials.refresh(request)
authed_session = AuthorizedSession(credentials)

def get_id():
    return "snapshot-" + time.strftime("%Y%m%d-%H%M%S")

def create_snapshot(request):
    trigger_run_url = "https://file.googleapis.com/v1beta1/projects/{}/locations/{}/instances/{}/snapshots?snapshotId={}".format(PROJECT_ID, SOURCE_INSTANCE_ZONE, SOURCE_INSTANCE_NAME, get_id())
    headers = {
      'Content-Type': 'application/json'
    }
    post_data = {
      "description": "test-instance snapshot",
    }
    print("Making a request to " + trigger_run_url)
    r = authed_session.post(url=trigger_run_url, headers=headers, data=json.dumps(post_data))
    data = r.json()
    if r.status_code == requests.codes.ok:
      print(str(r.status_code) + ": The snapshot is uploading in the background.")
      return
    else:
      raise RuntimeError(data['error'])

Once you have updated code file on local. Make sure that only these files exists on current location. Next is to create a cloud function, run the below command on same working directory:

gcloud functions deploy fs-snapshot \
  --runtime=python310 \
  --entry-point=create_snapshot \
  --trigger-http \
  --run-service-account=snapshotfunction@umeshkumhar-1.iam.gserviceaccount.com \
  --project=umeshkumhar-1

Cloud Function is successfully created with limited-access IAM policy. To enable Cloud Scheduler to invoke the cloud function grant iam policy binding as below:

gcloud functions add-iam-policy-binding fs-snapshot \
    --member serviceAccount:snapshotschedule@umeshkumhar-1.iam.gserviceaccount.com \
    --role roles/cloudfunctions.invoker --region=us-central1

3. Configure Cloud Scheduler to schedule the above Cloud Function using Cron expression.

In this step we will configure cloud scheduler to trigger cloud function using cron based schedule. For this example let’s consider we want to trigger snapshot every 30 minutes.

gcloud beta scheduler jobs create http fs-snapshot-schedule \
    --schedule "30 * * * *" \
    --http-method=GET \
    --uri=https://us-central1-umeshkumhar-1.cloudfunctions.net/fs-snapshot \
    --oidc-service-account-email=snapshotschedule@umeshkumhar-1.iam.gserviceaccount.com    \
    --oidc-token-audience=https://us-central1-umeshkumhar-1.cloudfunctions.net/fs-snapshot. \
    --location=us-central1 --project=umeshkumhar-1

And now your scheduler will invoke cloud function every 30 minutes and cloud function will trigger filestore snapshot.

This is how filestore snapshot will look like in GCP console.

Thanks for reading this blog on automating the filestore snapshot using Cloud Scheduler and Cloud Functions. Let me know if there is any feedback!

Don’t forget to checkout blog Part-III about setting up filestore snapshot retention.

GCP Filestore Snapshot Retention using Cloud Functions — Part III

If you have still not read the part I — check here

[How To] Snapshot GCP Filestore — Part I

Don’t forget to clap and follow on Medium 🙂 & also connect me on Linkedin.