End-to-End ML Workflow with GCP Cloud Functions

Cloud Functions is a serverless hosting option offered by the Google Cloud Platform (GCP), allowing users to deploy event-driven code without having to manage the underlying hardware.

With so many GCP products available, the question becomes: when should we use Cloud Functions, as opposed to other offerings such as App Engine, Compute Engine, Cloud Run, Kubernetes, etc.?

There are numerous scenarios that would merit using Cloud Functions — one big indicator that it would be helpful is if you want to set up a trigger between different GCP products.

Let’s take this example scenario: I want to run an object detection model every time I have a new set of images (animals in this case), which I upload manually to a GCS bucket. I would like the model (which will detect zebras) to run automatically when the images have been uploaded.

This can be done with the help of Cloud Functions, along with a GCS bucket (for our images) and a Compute Engine VM, where the model will be run via a startup script.

The architecture of this workflow will look like this:

As shown in the above diagram, Cloud Functions allows us to connect two GCP products together: a GCS bucket (which is the trigger for our Cloud Function) and a Compute Engine VM (which is what our Cloud Function triggers).

After our images have been manually uploaded to our GCS bucket (input) folder, our Cloud Function will start our Compute Engine instance. Once the instance is turned on, a startup script will a) mount our GCS bucket and b) run our model on the images in our bucket’s input folder (now mounted as a file system in the VM). When the model has finished running, it will write new images (those where zebras have been identified) to our bucket’s output folder. Our startup script then deletes our input image files, before unmounting our bucket.

Here’s how a step-by-step guide on how this was set up:

Step 1: Create Cloud Storage Bucket (the Trigger for Cloud Function)

Let’s create a Cloud Storage bucket where we will upload our input images and receive our output images. This bucket can be created in the console or via the command line:

gsutil mb -c standard -l us-west1 gs://animal-images-sg

And let’s add two folders: “input-images” and “output-images.”

Step 2: Configure VM (Target for Cloud Function)

The second step is to set up our VM and configure it to run our ML model.

I have chosen an n1-standard-4 machine with PyTorch already installed, with this boot disk:

The gcloud command to set up this specific instance is:

gcloud compute instances create zebra-classifier --zone=us-west1-b --machine-type=n1-standard-4 --scopes=https://www.googleapis.com/auth/cloud-platform --tags=http-server,https-server --image=c2-deeplearning-pytorch-1-4-cu101-20200414 --image-project=ml-images --boot-disk-size=100GB --boot-disk-type=pd-standard

Next, I cloned the object detection model repo from Github (git is pre-installed with this boot disk):

git clone https://github.com/stefangouyet/pytorch_objectdetecttrack

This code is forked from Chris Fotache’s object detection Github repository, which provides a great starting point for working with CNNs and the YOLO algorithm in PyTorch.

In order to download our YOLO weights, we first run this code:

cd config/
. download_weights.sh

Now, we install GCS Fuse:

export GCSFUSE_REPO=gcsfuse-`lsb_release -c -s`echo "deb 
http://packages.cloud.google.com/apt $GCSFUSE_REPO main" | sudo tee /etc/apt/sources.list.d/gcsfuse.listcurl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -sudo apt-get update
sudo apt-get install gcsfuse

After installing, let’s try mounting our animal-images-sg bucket, which will be mounted in a folder of the same name:

cd pytorch_objectdetecttrack/
mkdir animal-images-sg
gcsfuse --implicit-dirs animal-images-sg animal-images-sg/

Now that GCS Fuse is working, let’s try running our model on some input images.

I have added four images to the bucket’s input folder:

And after re-mounting the bucket, we can see the images in our shell:

FYI: To unmount the bucket, run:

fusermount -u animal-images-sg

Once the bucket is mounted and the new images are showing up, it’s time to run the object detection model, stored in the main.py file:

python3 main.py

The Python script outputs four new images (with “-det” appended to the filename). These are written to our output-images folder, and will show up in our GCS bucket as well:

Clicking on any of the output images will show the object detection model has correctly run, displaying the original image, plus a bounding box around the zebra.

We will go ahead and turn the VM off, and write our startup script:

#! /bin/sh

sleep 100
apt-get update
cd /home/gouyetarchitecture/pytorch_objectdetecttrack/

gcsfuse --implicit-dirs animal-images-sg animal-images-sg/
python3 main.py
rm animal-images-sg/input-images/*
fusermount -u animal-images-sg
sudo shutdown -h now

This startup script tells our VM to a) move to the current directory, b) mount our GCS Fuse bucket, c) run our object detection model, d) remove images in our input folder, e) unmount our bucket, and f) shutdown the VM. We will add the script to the VM’s Custom Metadata section, with a key of startup-script:

Step 3: Set up Cloud Function To Connect Storage and Compute Engine

Our final step involves using Cloud Functions, where we will connect it to both our GCS bucket and our Compute Engine instance.

First, we want our Event Type set as “Finalize/Create.” This tells Cloud Functions to wait for the creation or overwriting of an object in our bucket (more info can be found in GCP documentation).

Next, we specify which bucket we want to use (for our case, the animal-images-sg bucket).

Finally, we declare what we want to occur when our Cloud Function is triggered. This can be defined in several different languages — usually, I do this using Node.js (version 8 in this case):

var http = require('http');
var Compute = require('@google-cloud/compute');
var compute = Compute();exports.startInstance = function startInstance(req, res) {
var zone = compute.zone('us-west1-b');
var vm = zone.vm('zebra-classifier');
vm.start(function(err, operation, apiResponse) {
console.log('instance start successfully');
});
res.status(200).send('Success start instance');};

The above index.js file contains a function startInstance that will look for our zebra-classifier instance in zone us-west1-b and calls the built-in vm.start function.

And the package.json file can remain with its default configuration:

Summary:

And that’s the full setup!

We can now see our end-to-end workflow:

When we upload new images to our GCS bucket, our VM starts up automatically:

Uploading images in our bucket turns on our zebra-classifier VM instance

We’ve uploaded 10 images above, with seven of them zebras. After waiting a couple minutes, we will find an empty input folder and an output folder with the seven images of zebras, with a bounding box around the animal:

Input images are deleted and the output images are now visible

And we can detect which of our images contain zebras!

I hope you enjoyed reading this. If you have comments or suggestions, please feel free to contact me by leaving a comment below. Thanks!