Google Cloud Storage “exploder” #2

Golang Cloud Functions

Gophers and Explosions!?

Reference: Hallmark Caddyshack Gopher Ornament

Not quite but I’m still alright…

Almost one year ago, I wrote a proof-of-concept Node.JS Cloud Function (is that the correct singular form?) that unzipped files triggered by uploaded to GCS: exploder.

With the availability of Golang Cloud Functions Alpha (early access) and inspired by JBD post (link), herewith a simple (single-threaded) rewrite in Golang.

Setup

PROJECT=[[YOUR-PROJECT]]
BILLING=[[YOUR-BILLING]]
BUCKET_SRC=[[YOUR-SOURCE-BUCKET]]
BUCKET_DST=[[YOUR-OUTPUT-BUCKET]]
gcloud projects create ${PROJECT}
gcloud beta billing projects link ${PROJECT} \
--billing-account=${BILLING}
gcloud services enable cloudfunctions.googleapis.com \
--project=${PROJECT}
gsutil mb -c regional -l us-west1 -p ${PROJECT} gs://${BUCKET_SRC}
gsutil mb -c regional -l us-west1 -p ${PROJECT} gs://${BUCKET_DST}
# Go [as usual]
mkdir -p go
export GOPATH=${PWD}/go
go get -u golang.org/x/vgo
# Vgo
mkdir -p ${PWD}/vgo/exploder
cd vgo/exploder
touch exploder.go # Copy contents from file below
touch go.mod
vgo build && \
go mod vendor && \
gcloud alpha functions deploy Exploder \
--entry-point=Exploder \
--runtime go111 \
--set-env-vars=BUCKET_DST=${BUCKET_DST} \
--trigger-resource=${BUCKET_SRC} \
--trigger-event=google.storage.object.finalize \
--project=${PROJECT}

And:

Then, find yourself a zipped file, upload it to ${BUCKET_SRC}, slurp coffee (red wine, depending on the time of day), then check ${BUCKET_DST}:

gsutil cp ${FILE} gs://${BUCKET_SRC}
gsutil ls -r gs://${BUCKET_DST}

Or using Cloud Console’s Storage Browser:

Source

And:

Destination

And then, slightly more involved:

tree x
x
├── a
│   └── IMG_13281.jpg
├── b
│   └── IMG_13281.jpg
└── c
    ├── c1
    │   └── IMG_13281.jpg
    └── c2
        └── IMG_13281.jpg

And:

gsutil ls -r gs://${BUCKET_DST}
gs://dazwilkin-180903-dst/x/a/:
gs://dazwilkin-180903-dst/x/a/IMG_13281.jpg
gs://dazwilkin-180903-dst/x/b/:
gs://dazwilkin-180903-dst/x/b/IMG_13281.jpg
gs://dazwilkin-180903-dst/x/c/:
gs://dazwilkin-180903-dst/x/c/c1/:
gs://dazwilkin-180903-dst/x/c/c1/IMG_13281.jpg
gs://dazwilkin-180903-dst/x/c/c2/:
gs://dazwilkin-180903-dst/x/c/c2/IMG_13281.jpg

Update: 2018–09–05 — Added OpenCensus

Since JBD wrote about using OpenCensus, I thought I’d revisit this story, the code and add a stat for the count of unzipped files (link) and a trace for the GCS object creates.

Stackdriver Metric Explorer

And some results though I’m perplexed by the apparent low rate :-(

The trace was more problematic for me to get working but I think it was because I was not reliably vgo build && go mod vendor between deployments. So, please, bear that in mind as you develop code.

Stackdriver Trace

The coding changes are trivial.

Conclusion

The Golang runtime for Cloud Functions is a compelling (albeit overdue) addition to the set of runtimes. One noteworthy benefit with Golang over Node.JS and Python is that your Functions may be multi-threaded. This example is a mostly direct transliteration of the Node.JS example. A good evolution of this would be make this code multi-threaded (and async).

That’s all!