Workflows state management with Firestore

In Workflows, sometimes, you need to store some state, a key/value pair, in a step in one execution and later read that state in another step in another execution. There’s no intrinsic key/value store in Workflows. However, you can use Firestore as a key/value store and that’s what I want to show you here.

If you want to skip to see some samples, check out workfow.yaml.

If you want to learn more about it, keep reading.

Firestore setup for Workflows

In Firestore, you typically have a single collection and multiple documents in that collection. To use Firestore as a key/value store for Workflows, one idea is to use the workflow name as the collection name and use a single document to store all the key/value pairs.

your-workflow-name
|
├── key-value-store
|
├── key1=value1
├── key2=true
├── key3=1
├── key4=1.5

Put value

Here’s a subworkflow to save a key/value pair on Firestore:

firestore_put:
params: [key, value, valueType: "string"]
steps:
- init:
assign:
- database_root: ${"projects/" + sys.get_env("GOOGLE_CLOUD_PROJECT_ID") + "/databases/(default)/documents/" + sys.get_env("GOOGLE_CLOUD_WORKFLOW_ID") + "/"}
- doc_name: ${database_root + "key-value-store"}
- store:
call: googleapis.firestore.v1.projects.databases.documents.patch
args:
name: ${doc_name}
updateMask:
fieldPaths: [${key}]
body:
fields:
${key}:
${valueType + "Value"}: ${value}

In Firestore, you need to specify the type of the variable being saved. The subworkflow assumes the string value type but you can also pass in some other basic types such as boolean, integer, double.

Get value

Here’s the subworkflow to retrieve the value for a given key:

firestore_get:
params: [key, valueType: "string"]
steps:
- init:
assign:
- database_root: ${"projects/" + sys.get_env("GOOGLE_CLOUD_PROJECT_ID") + "/databases/(default)/documents/" + sys.get_env("GOOGLE_CLOUD_WORKFLOW_ID") + "/"}
- doc_name: ${database_root + "key-value-store"}
- get:
call: googleapis.firestore.v1.projects.databases.documents.get
args:
name: ${doc_name}
mask:
fieldPaths: [${key}]
result: getResult
- return_value:
switch:
- condition: ${not("fields" in getResult)}
return: null
- condition: true
return: ${getResult.fields[key][valueType + "Value"]}

Note that if the key does not exist, the subworkflow simply returns null. I thought this is easier than throwing a KeyNotFound error and let the user handle it.

Cleanup

You also probably want to clear the keys once in a while by deleting the document. Here’s a subworflow for that:

firestore_clear:
steps:
- init:
assign:
- database_root: ${"projects/" + sys.get_env("GOOGLE_CLOUD_PROJECT_ID") + "/databases/(default)/documents/" + sys.get_env("GOOGLE_CLOUD_WORKFLOW_ID") + "/"}
- doc_name: ${database_root + "key-value-store"}
- drop:
call: googleapis.firestore.v1.projects.databases.documents.delete
args:
name: ${doc_name}

Examples

Now that subworkflows are in place, this is how you can store key/value pairs for different types and retrieve results. Notice the optional clear_keys step in the end:

main:
steps:
- put_string_value:
call: firestore_put
args:
key: "key1"
value: "value1"
- get_string_value:
call: firestore_get
args:
key: "key1"
result: string_value
- put_boolean_value:
call: firestore_put
args:
key: "key2"
value: true
valueType: "boolean"
- get_boolean_value:
call: firestore_get
args:
key: "key2"
valueType: "boolean"
result: boolean_value
- put_integer_value:
call: firestore_put
args:
key: "key3"
value: 1
valueType: "integer"
- get_integer_value:
call: firestore_get
args:
key: "key3"
valueType: "integer"
result: integer_value
- put_double_value:
call: firestore_put
args:
key: "key4"
value: 1.5
valueType: "double"
- get_double_value:
call: firestore_get
args:
key: "key4"
valueType: "double"
result: double_value
- get_nonexisting_key:
call: firestore_get
args:
key: "nonexisting"
result: nonexisting_value
# - clear_keys:
# call: firestore_clear
- return_values:
return:
string_value: ${string_value}
boolean_value: ${boolean_value}
integer_value: ${integer_value}
double_value: ${double_value}
nonexisting_value: ${nonexisting_value}

Even though Workflows does not provide an intrinsic key/value store, Firestore is quite easy to use instead once you figure out the right API calls.

What do you think of this solution? If you have ideas to improve it, please reach out to me on Twitter @meteatamel.

Originally published at https://atamel.dev.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store