Google Cloud Storage “downsizer”

Further adventures with Google Cloud Functions

Published in

Google Cloud - Community

5 min readDec 21, 2017

Nobody stop me… this is too much fun ;-)

App Engine Standard provides an Images service that many customers value and often use to create thumbnails (in Google Cloud Storage). Firebase has a Cloud Functions sample that uses ImageMagick for thumbnails too.

Here’s my version. It uses file streaming so should (haven’t tested) scale better and it creates multiple downsized (thumbnail) images. I decided to use Cloud Source Repositories to host my code which works well though would benefit from change triggering of new Cloud Functions deployments… files FR.

Setup

I think you may benefit from Google Cloud Platform’s Free (as in beer) Tier because Google Cloud Functions, Google Cloud Storage and Google Cloud Source Repositories appear all to be included.

Open a bash terminal and get coding:

ROOT=$(whoami)-171220
BILLING=[[YOUR-BILLING-ID]]
PROJECT=[[YOUR-PROJECT-ID]] // ${ROOT}-downsizer
REGION=us-central1# Create GCP Project
gcloud alpha projects create $PROJECT# Link it to your Billing Account
gcloud alpha billing projects link $PROJECT --account-id=$BILLING# Enable Cloud Functions
gcloud services enable cloudfunctions.googleapis.com \
--project=$PROJECT# GCS is enabled by default create 2 buckets
for BUCKET in trigger thumbnails
do
  gsutil mb \
  -c regional \
  -l ${REGION} \
  -p ${PROJECT} \
  gs://${ROOT}-${BUCKET}
done

Cloud Source Repositories (CSR)

CSR is a service that provides hosted, private Git repos. I’m generally lazy and use a regular file system for hosting my code but, keeping changes is great with Git. Once the CSR repo is created, clone it locally for your code:

# Create a Cloud Source Repo called "default"
gcloud source repos create default# Change to your working directory
gcloud source repos clone default --project=${PROJECT}

Create index.js:

And package.json:

Open Visual Studio Code (or your preferred editor). Replace the value of ‘root’ (#10) with the value of your bash env variable ${ROOT}. Save it.

Visual Studio Code includes a Git client and so you may stage your changes and define a commit using it. Alternatively, if you prefer to use the command-line:

git add index.js
git add package.json
git commit --message "Initial commit"
git push -u origin master

Any time you change either of these files, repeat the add, commit, push command to ensure your files are reflected correctly in CSR:

https://console.cloud.google.com/code/develop/browse/default/master?project=${PROJECT}

Let’s deploy this code as a Cloud Function

Cloud Functions

https://console.cloud.google.com/functions/list?project=${PROJECT}

Sticking with the command-line, you should be able to:

gcloud beta functions deploy downsizer \
--source https://source.developers.google.com/projects/${PROJECT}/repos/default/moveable-aliases/master/paths/ \
--trigger-bucket=${ROOT}-trigger \
--entry-point=thumbnail \
--project=$PROJECT

You’ll note that I’m mixing up “downsizer” and “thumbnail”, apologies. I started with “thumbnail” but now prefer “downsizer”. It’s more… dramatic. Fortunately, Cloud Functions permits aliasing. We give Cloud Functions the name “downsizer” but we tell it that the function exported in the Node.js code is called “thumbnail”. The value of the source flag is specific to ${PROJECT}, the fact that the repo is called “default” and that we’re using “master” for simplicity.

A few minutes later…

Deploying function (may take a while - up to 2 minutes)...

Or, if you’d prefer:

gcloud beta functions list --project=$PROJECT
NAME       STATUS  TRIGGER        REGION
downsizer  ACTIVE  Event Trigger  us-central1

Testing

Because we created buckets for this project and have yet to use them, they’re both empty:

gsutil ls gs://${ROOT}-trigger
gsutil ls gs://${ROOT}-thumbnails

Find your favorite image and use gsutil to copy it to the ‘trigger’ bucket. This will trigger the Cloud Function to generate 4 thumbnails of our image in the ‘thumbnail’ bucket. In my example, I’m also moving (renaming) the file to “/2013/road-trip/henry.jpg” to show that the source path is preserved.

gsutil cp henry.jpg gs://${ROOT}-trigger/2013/road-trip
Copying file:.../henry.jpg [Content-Type=image/jpeg]...
- [1 files][  1.7 MiB/  1.7 MiB]                         
Operation completed over 1 objects/1.7 MiB.

Here’s me with my best buddy:

This image is 4000x3000 pixels and is 1.7 MiB

All being well, you should expect 4 thumbnails to appear in the ‘thumbnails’ bucket. Here’s my result:

gsutil ls gs://${ROOT}-thumbnails/2013/road-trip/
gs://${ROOT}-thumbnails/2013/road-trip/henry.jpg_128x128
gs://${ROOT}-thumbnails/2013/road-trip/henry.jpg_256x2256
gs://${ROOT}-thumbnails/2013/road-trip/henry.jpg_64x64

Enumerate these using “gsutil ls -l” I learn that they are 18364, 26154, and 15758 bytes per the list order which sounds about right. So, let’s confirm:

The 64x64 is actually 64 by 48 because my original image was not square (64/4000*3000=48)

64x48

And, 256x256 is actually 256 by 192 for the same reason:

Logs

I’ve become slightly obsessive recently about grepping Stackdriver Logging logs so, I’ll control myself and just give you an example:

gcloud logging read "resource.type=\"cloud_function\" resource.labels.function_name=\"downsizer\"" \
--project=$PROJECT \
--freshness=10m \
--format='value(textPayload)'

NB the “freshness” flag which I learned of earlier today and, in this case, just retrieves the last 10 minutes’ logs.

Update: 21-Dec-17

I revised the sample to leverage header metadata to specify the desired thumbnail sizes. The code above reflects this. Now, if the object includes a ‘goog-meta-sizes’ header with an array of sizes, these will used instead of the default set of [“256x256”,”128x128",”64x64"]:

gsutil  \
-h 'x-goog-meta-sizes:["8x8","32x32","512x512"]' \
cp henry.jpg gs://{ROOT}-trigger/2013/road-trip/beach/

Results in:

gsutil ls gs://${ROOT}-thumbnails/2013/road-trip/beach/
gs://${ROOT}-thumbnails/2013/road-trip/beach/henry.jpg_32x32
gs://${ROOT}-thumbnails/2013/road-trip/beach/henry.jpg_512x512
gs://${ROOT}-thumbnails/2013/road-trip/beach/henry.jpg_8x8

NB I find it difficult to track down the specifications for Cloud Functions event data. It is documented, here and then, for Cloud Storage Objects here.

Conclusion

This isn’t vastly different to my earlier Exploder post but, absent alternatives from Google or 3rd-parties, it’s straightforward to build, deploy and run, Cloud Functions for event-driven processing.

Feedback always welcome!

Tidy-up

You can delete the Cloud Functions:

gcloud beta functions delete downsizer \
--project=${PROJECT} \
--quiet

You may delete buckets after recursively delete all their objects. Please be VERY careful using this command:

gsutil rm -r gs://${ROOT}-trigger
gsutil rm -r gs://${ROOT}-thumbnails

Alternatively you can simply delete the project which will delete everything within it too:

gcloud projects delete ${PROJECT} --quiet

Thanks!