Day #16 with Workflows: Reading in and writing a JSON file to a storage bucket from a workflow

Workflows provides several connectors for interacting with various Google Cloud APIs and services. In the past, I’ve used for example the Document AI connector to parse documents like expense receipts, or the Secret Manager connector to store and access secrets like passwords. Another useful connector I was interested in using today was the Google Cloud Storage connector, to store and read files stored in storage buckets.

Those connectors are auto-generated from their API discovery descriptors, but there are some limitations currently that prevent, for example, to download the content of a file. So instead of using the connector, I looked at the JSON API for cloud storage to see what it offered (insert and get methods).

What I wanted to do was to store a JSON document, and to read a JSON document. I haven’t tried with other media types yet, like pictures or other binary files. Anyhow, here’s how to write a JSON file into a cloud storage bucket:

main:
params: [input]
steps:
- assignment:
assign:
- bucket: YOUR_BUCKET_NAME_HERE
- write_to_gcs:
call: http.post
args:
url: ${"https://storage.googleapis.com/upload/storage/v1/b/" + bucket + "/o"}
auth:
type: OAuth2
query:
name: THE_FILE_NAME_HERE
body:
name: Guillaume
age: 99

In the file, I’m storing a JSON document that contains a couple keys, defined in the body of that call. By default, here, a JSON media type is assumed, so the body defined at the bottom in YAML is actually written as JSON in the resulting file. Oh and of course, don’t forget to change the names of the bucket and the object in the example above.

And now, here’s how you can read the content of the file from the bucket:

main:
params: [input]
steps:
- assignment:
assign:
- bucket: YOUR_BUCKET_NAME_HERE
- name: THE_FILE_NAME_HERE
- read_from_gcs:
call: http.get
args:
url: ${"https://storage.googleapis.com/download/storage/v1/b/" + bucket + "/o/" + name}
auth:
type: OAuth2
query:
alt: media
result: data_json_content
- return_content:
return: ${data_json_content.body}

This time we change the GCS URL from “upload” to “download”, and we use the alt=media query parameter to instruct the GCS JSON API that we want to retrieve the content of the file (not just its metadata). In the end, we return the body of that call, which contains the content.

Originally published at https://glaforge.appspot.com.

--

--

--

A collection of technical articles and blogs published or curated by Google Cloud Developer Advocates. The views expressed are those of the authors and don't necessarily reflect those of Google.

Recommended from Medium

7 Myths/Misconception Of A Programming Language.

We share, we listen, we improve!

Interview Question 10 - A Coding Concept

11 Reasons Why Software Engineering Isn’t a Stressful Job

Some Details Regarding Our Organizational Structure

Where is Agile Headed?

Publish the “google-services.json” file of an android app project or not?

Python for kids

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Guillaume Laforge

Guillaume Laforge

Developer Advocate for Google Cloud Platform, Apache Groovy programming language project VP/Chair

More from Medium

Building APIs with Cloud Functions and API Gateway

Serverless APIs made simple on GCP with Goblet backed by Cloud Functions and Cloud Run

GCP integration with PagerDuty using Terraform

Use Case Business Account Insight API: How SMSParking decreases fraud and improves the customer…