TECHNICAL BLOG / GOOGLE CLOUD PLATFORM

GCP Filestore Snapshot Retention using Cloud Functions — Part III

This blog is about scheduling the cloud function that can perform filestore snapshot retention as per the requirements. Currently such feature is not supported in filestore out of the box.

If you have not gone through Part I & II which includes basics around Filestore Snapshots and how to perform snapshots using Cloud scheduler and Cloud Functions.

The implementation is very much similar to how scheduling filestore snapshot works as documented in part — II. I would suggest to read that before going forward.

[How To] Schedule Filestore Snapshot using Cron — Part II

Design Flow

How the flow would look like:

1. Cloud Scheduler service will be configured using same cron expression and snapshot retention cloud function. As the schedule triggers the Cloud Scheduler would invoke execution of mentioned target Cloud Function.
2. Cloud Function service will use gcloud apis to list out all snapshots of the filestore instance and perform retention job.

Logic Breakthrough

Here we will understand how exactly the snapshot retention is taken care of in cloud function code.

The code snippet will take following inputs:

• snapshot retention in hours
• project ID
• filestore instance name
• filestore instance location

The snapshot retention hours is the number of hours that each filestore snapshot should retain. That means calculating from creation time to current time should not exceed more than snapshot retention hours, else those snapshot will be deleted.

For example, we have considered that cloud function will run as nightly job and snapshot retention hours is configured as 120 hours. When the cloud function executes, it list down all snapshots of defined filestore and calculate the snapshot time from their creation timestamp and compares with defined retention hours. If snapshot time exceeds retention hours then that snapshot will be deleted and only latest 5 days snapshot will be retained.

Implementation Flow

We will deploy the complete design flow into 3 steps as we performed in part-II and here we will just focus on the cloud function and how filestore snapshot retention can be achieved.

Configure Cloud Function with the filestore snapshot retention code.

In this step, we will create a cloud function named fs-snapshot-retention with python code in 3.10. Before deploying the cloud function, create python file named main.py and copy below code that read variable values and accordingly trigger the filestore snapshot.

Requirements: requirements.txt

google-auth==1.19.2
requests==2.24.0

Code Snippet: main.py

PROJECT_ID = 'umeshkumhar-1'
SOURCE_INSTANCE_ZONE = 'us-central1'
SOURCE_INSTANCE_NAME = 'test-instance'
SNAPSHOT_RETENTION_TIME_HRS = 120 
# 5 days retention (24 * 5 = 120)

import google.auth
import google.auth.transport.requests
from google.auth.transport.requests import AuthorizedSession
import time
import requests
import json

credentials, project = google.auth.default()
request = google.auth.transport.requests.Request()
credentials.refresh(request)
authed_session = AuthorizedSession(credentials)

now = time.time()
retention_seconds = SNAPSHOT_RETENTION_TIME_HRS * 60 * 60

def delete_snapshot(request):
    list = []
    trigger_run_url = "https://file.googleapis.com/v1beta1/projects/{}/locations/{}/instances/{}/snapshots".format(PROJECT_ID, SOURCE_INSTANCE_ZONE, SOURCE_INSTANCE_NAME)
    r = authed_session.get(trigger_run_url)
    data = r.json()
    if not data:
        print("No snapshots to delete.")
    else:
        list.extend(data['snapshots'])
        while 'nextPageToken' in data.keys():
            nextPageToken = data['nextPageToken']
            trigger_run_url_next = "https://file.googleapis.com/v1beta1/projects/{}/locations/{}/instances/{}/snapshots?pageToken={}".format(PROJECT_ID, SOURCE_INSTANCE_ZONE, SOURCE_INSTANCE_NAME, nextPageToken)
            r = authed_session.get(trigger_run_url_next)
            data = r.json()
            list.extend(data['snapshots'])
    for i in list:
        snap_time = i['createTime']
        snap_time = snap_time[:-4]
        snap_time = float(time.mktime(time.strptime(snap_time, "%Y-%m-%dT%H:%M:%S.%f")))
        if now - snap_time > retention_seconds:
            print("Deleting " + i['name'] + " in the background.")
            r = authed_session.delete("https://file.googleapis.com/v1beta1/{}".format(i['name']))
            data = r.json()
            if r.status_code == requests.codes.ok:
              print(str(r.status_code) + ": Deleting " + i['name'] + " in the background.")
            else:
              raise RuntimeError(data['error'])

Once you have updated code file on local. Make sure that only these files exists on current location. Next is to create a cloud function, run the below command on same working directory:

gcloud functions deploy fs-snapshot-retention \
  --runtime=python310 \
  --entry-point=delete_snapshot \
  --trigger-http \
  --run-service-account=snapshotfunction@umeshkumhar-1.iam.gserviceaccount.com \
  --project=umeshkumhar-1

Cloud Function is successfully created and ready to perform snapshot deletion tasks.

Thanks for reading this blog on automating the filestore snapshot retention using Cloud Scheduler and Cloud Functions. Let me know if there is any feedback!

If you have still not read the part I — check here & part II — check here

[How To] Snapshot GCP Filestore — Part I

[How To] Schedule Filestore Snapshot using Cron — Part II

Don’t forget to clap and follow on Medium 🙂 & also connect me on Linkedin.