Google Cloud Storage “exploder” #2

Golang Cloud Functions

Daz Wilkin
Sep 3, 2018 · 3 min read

Gophers and Explosions!?

Reference: Hallmark Caddyshack Gopher Ornament

Not quite but I’m still alright…

Almost one year ago, I wrote a proof-of-concept Node.JS Cloud Function (is that the correct singular form?) that unzipped files triggered by uploaded to GCS: exploder.

With the availability of Golang Cloud Functions Alpha (early access) and inspired by JBD post (link), herewith a simple (single-threaded) rewrite in Golang.

Setup

PROJECT=[[YOUR-PROJECT]]
BILLING=[[YOUR-BILLING]]
BUCKET_SRC=[[YOUR-SOURCE-BUCKET]]
BUCKET_DST=[[YOUR-OUTPUT-BUCKET]]
gcloud projects create ${PROJECT}gcloud beta billing projects link ${PROJECT} \
--billing-account=${BILLING}
gcloud services enable cloudfunctions.googleapis.com \
--project=${PROJECT}
gsutil mb -c regional -l us-west1 -p ${PROJECT} gs://${BUCKET_SRC}
gsutil mb -c regional -l us-west1 -p ${PROJECT} gs://${BUCKET_DST}
# Go [as usual]
mkdir -p go
export GOPATH=${PWD}/go
go get -u golang.org/x/vgo
# Vgo
mkdir -p ${PWD}/vgo/exploder
cd vgo/exploder
touch exploder.go # Copy contents from file below
touch go.mod
vgo build && \
go mod vendor && \
gcloud alpha functions deploy Exploder \
--entry-point=Exploder \
--runtime go111 \
--set-env-vars=BUCKET_DST=${BUCKET_DST} \
--trigger-resource=${BUCKET_SRC} \
--trigger-event=google.storage.object.finalize \
--project=${PROJECT}

And:

Then, find yourself a zipped file, upload it to ${BUCKET_SRC}, slurp coffee (red wine, depending on the time of day), then check ${BUCKET_DST}:

gsutil cp ${FILE} gs://${BUCKET_SRC}
gsutil ls -r gs://${BUCKET_DST}

Or using Cloud Console’s Storage Browser:

Source

And:

Destination

And then, slightly more involved:

tree x
x
├── a
│ └── IMG_13281.jpg
├── b
│ └── IMG_13281.jpg
└── c
├── c1
│ └── IMG_13281.jpg
└── c2
└── IMG_13281.jpg

And:

gsutil ls -r gs://${BUCKET_DST}
gs://dazwilkin-180903-dst/x/a/:
gs://dazwilkin-180903-dst/x/a/IMG_13281.jpg
gs://dazwilkin-180903-dst/x/b/:
gs://dazwilkin-180903-dst/x/b/IMG_13281.jpg
gs://dazwilkin-180903-dst/x/c/:
gs://dazwilkin-180903-dst/x/c/c1/:
gs://dazwilkin-180903-dst/x/c/c1/IMG_13281.jpg
gs://dazwilkin-180903-dst/x/c/c2/:
gs://dazwilkin-180903-dst/x/c/c2/IMG_13281.jpg

Update: 2018–09–05 — Added OpenCensus

Stackdriver Metric Explorer

And some results though I’m perplexed by the apparent low rate :-(

The trace was more problematic for me to get working but I think it was because I was not reliably vgo build && go mod vendor between deployments. So, please, bear that in mind as you develop code.

Stackdriver Trace

The coding changes are trivial.

Conclusion

That’s all!

Google Cloud Platform - Community

A collection of technical articles published or curated by Google Cloud Platform Developer Advocates. The views expressed are those of the authors and don't necessarily reflect those of Google.

 by the author.

Daz Wilkin

Written by

Google Cloud Platform - Community

A collection of technical articles published or curated by Google Cloud Platform Developer Advocates. The views expressed are those of the authors and don't necessarily reflect those of Google.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade