100 Followers
·
Follow

Mastering Google Cloud Build Config Syntax

Say the magic words, and GCB opens up with hidden potential

Image for post
Image for post

Google’s Cloud Build automation platform is modeled on a series of containerized “builder steps, specified in a YAML config file. For each step, GCB spins up a Docker container, and runs a specific command within the context of that container, like gradle, or gcloud, or wget. You can pass one or more arguments to that command using the args field in your cloudbuild.yaml file.

This syntax makes it easy to get a basic build pipeline up and running. But as you add complexity to your pipeline, you may find that your YAML becomes hard to parse and maintain, or even that there are things you need to do that simply can’t be expressed. In this post, we’ll explore alternative ways to write your Cloud Build config. By applying these best practices, you’ll have more readable, maintainable config files. Even better, you can harness the hidden power of Cloud Build to create advanced CI/CD pipelines.

Image for post
Image for post
These config files all do the exact same thing. Which one is best? That depends on your use case.

As an example, we’ll make a build step that fetches text from a remote API, and prints it to the console. Let’s start with the simplest version of the config:

Simple Syntax

steps:
- name: 'gcr.io/cloud-builders/curl'
args: ['https://pets.doingdevops.com/pet','-s','--max-time','10']

This is the most common syntax found in documentation, and it’s usually the right place to start.

Best Practice: Use this for simple jobs: it’s concise and easy to read.

Expanded syntax (collection style)

steps:
- name: 'gcr.io/cloud-builders/curl'
args:
[
'https://pets.doingdevops.com/pet',
'-s',
'--max-time', '10', # related args on one line
]

By adding line breaks, we make room for long arguments, and make it easier to see where one argument ends and the next one begins. This is especially helpful when using variables or data structures within arguments, which can easily become hard to read.

However, the one-argument-per-line rule isn’t actually enforced. So, if multiple arguments are related, they can be grouped one line. In this example, the arguments --max-timeand 10are on the same line, to make it evident that one refers to the other.

Best Practice: Use this syntax whenever basic syntax becomes cumbersome: many arguments, long arguments, complexity within arguments, etc. And don’t forget to add a dangling comma!

Expanded syntax (list style)

steps:
- name: 'gcr.io/cloud-builders/curl'
args:
- 'https://pets.doingdevops.com/pet'
- '-s'
- '--max-time'
- '10'

While this is similar to “Expanded syntax (collection style),” it has a few disadvantages: 1) having a dash per argument reduces readability, especially since many arguments will have their own dashes; 2) you can’t group multiple arguments on a single line; 3) it’s harder to convert from “Simple syntax.”

Best Practice: Don’t use. (Prefer “collection style.”)

Breakout syntax

But we’re not limited only to the default command: we can “break out,” and run any command available in the container, simply by specifying an alternate entrypoint. Most builders have bash installed, which gives us ultimate flexibility to do anything we wish within our step. Here’s the same operation, but invoking bash directly:

steps:
- name: 'gcr.io/cloud-builders/curl'
entrypoint: 'bash'
args:
- '-c' # pass what follows as a command to bash
- |
curl -s 'https://pets.doingdevops.com/pet' --max-time 10

Here, a block YAML syntax is used (the argument is prefaced with “|”), so we can pass a multiline unquoted string as the command. And what exactly have we achieved? Nothing! This still does the same thing as the “simple” example. But now we have the full power to run arbitrary bash commands. So we can do more. Much more…

Let’s suppose that the remote API is flaky; sometimes it fails. In that case, we want to retry until it succeeds. Here’s an example that uses multiple commands and conditional logic to ensure that we get a valid response:

steps:
- name: 'gcr.io/cloud-builders/curl'
entrypoint: 'bash'
args:
- '-c'
- |
PET="$$(curl -s https://pets.doingdevops.com/pet_flaky --max-time 10)"
while [ "$$PET" == "ERROR" ] ; do
echo "Error: API failed to respond with a pet! Try again..."
PET="$$(curl -s https://pets.doingdevops.com/pet_flaky --max-time 10)"
done
echo "Success! $${PET}"

Note the use of double dollar signs ($$) as escape characters. This ensures that Cloud Build will pass them to the container, rather than interpreting them as substitutions.

This is more than just a one-off curl command! By overriding the entry point, and stringing together bash commands, we can do anything that’s possible within the confines of this builder container. And if you need to do something that’s not available in an existing builder, you can make your own custom builder image — be sure to include a shell (e.g. bash) if you plan to use “breakout” syntax.

What’s something interesting you’ve done with Cloud Build? Leave a note in the comments!

Example config files, and source for the remote API services, can be found at github.com/davidstanke/gcb-syntax-blog

Written by

DevOps Advocate at Google. My home is in Jersey City. My office is in New York. My opinions are all over the map, and may not reflect those of my employer.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store