Overcoming Common Challenges of Google Cloud Workflows

Image by rawpixel.com on Freepik

Google Cloud Workflows is a robust service that provides a potent platform for event processing and batch jobs. It is highly scalable and easily monitored, making it an ideal choice for businesses seeking a cost-effective solution.

At DeliveryHero Tech Hub, it has been more than a year since we first implemented Google Cloud Workflows in our projects. Since then, we have seamlessly orchestrated over 30 workflows, utilizing the service to its fullest potential.

Over time, we have developed several methodologies and workarounds to address the challenges we encountered while developing our 30+ Workflows code. These include strategies for managing environment variables, syntax validation, testing, complexity, and resource limitations.

Challenges are what make life interesting; overcoming them is what makes life meaningful. — Joshua J. Marine

Environment Variables

The usage of multiple environments of software projects is common practice, so we need to define environment variables to deploy the same workflows definition to multiple environments.

Unfortunately, Workflows does not support user-defined environment variables as of now as mentioned in documentation.

User-defined environment variables are not supported.

Google suggests using a placeholder for variables and replacing them on the deployment pipeline as a workaround on their workflows-demos repository.

A sample taken from demos repo:

main:
  steps:
    - init:
        assign:
          - url1: REPLACE_url1
          - url2: REPLACE_url2

steps:
- id: 'replace-urls'
  name: 'gcr.io/cloud-builders/gcloud'
  entrypoint: bash
  args:
    - -c
    - |
      sed -i -e "s~REPLACE_url1~$_URL1~" workflow2.yaml
      sed -i -e "s~REPLACE_url2~$_URL2~" workflow2.yaml
- id: 'deploy-workflow'
  name: 'gcr.io/cloud-builders/gcloud'
  args: ['workflows', 'deploy', 'multi-env2-$_ENV', '--source', 'workflow2.yaml']

We use the same approach on our deployment pipelines. It creates additional steps to the deployment pipeline but it is considerable.

Syntax Validation

Workflows utilize both JSON and YAML to define steps and execution logic. As it relies on a special schema, it’s necessary to validate the syntax of certain steps before deployment. GCP Console offers a visually supported, sophisticated editor that includes syntax checks for. However, gcloud CLI currently lacks a command to perform this operation.

JSON Schema store has a schema for Cloud Workflows but it still raises some false positives for valid Workflow YAML files.

Valid workflow with false positive (Incorrect type. Expected ”number”)

To ensure syntax validity, we have found that deploying workflows to a sandbox environment is the safest option.

Testing

Having integration and unit tests is the safest and cheapest way to test your code/configuration. Unfortunately, Cloud Workflows does not provide a built-in testing mechanism or does not support running on local machines. However, it is possible to create mock third-party resources and execute workflows with test payloads by utilizing mock API calls for testing purposes.

We have still that task on our roadmap but probably preparing that kind of mock services would take time.

Complexity

Thanks to calls and controlling flow elements like Jumps, Conditions, Subworkflows, it’s possible to embed logic and increase complexity in your Workflow.

While Workflows is not a programming language and lacks certain capabilities, it is still possible to include complex logic within the Workflow. However, it is generally advisable to keep business logic within APIs, as they are better equipped to handle such tasks.

In order to maintain clarity and reduce the likelihood of errors, setting a low cyclomatic complexity target for your workflows would be a good practice.

Resource Limitations

Workflows is generous on quotas, step limits, but not on resource limits. It’s easy to hit memory limitations, especially the ones below.

• 2MB — Response size: The maximum size of an HTTP response (if saved to a variable, the memory limit for variables applies)
• 512 KB — Memory limit for variables: variables: The maximum amount of memory that you can use for all variables that you define in a single execution of a given workflow

As it’s not possible to expand these limitations, your best option is to utilize filtering and perform processing on the API level.

Internal API Access / Private VPC

When utilizing the HTTP connector in Workflows, it is only possible to invoke external endpoints, not internal ones that attached to a VPC. (By the way, you can access Cloud Run or Cloud Function with ingress settings: Allow internal traffic only).

As guillaume blaquiere mentioned on StackOverflow post, it is one of the most demanding features of Workflows for now, it has yet to be implemented.

Google Cloud Workflow: Reach Private VPC

You’ll need a proxy your internal endpoints with Cloud Run or Cloud Function (with internal ingress) when you need that kind of access.

It is not so elegant way but at least we have a workaround :)

Summary

These are the most common challenges that we faced while developing or maintaining Workflows over time. Most of them are related to the limitations of the current implementation and hopefully will have solutions in future releases. But we shouldn’t expect Workflows to be a programming language.

Don’t forget to read Guillaume Laforge’s Workflows tips’n tricks article, as it contains valuable tips and tricks!

Workflows tips’n tricks

Please don’t hesitate to leave comments if you have any more refined solutions to the challenges.