CloudFormation coding using YAML

This is not a “101” on CloudFormation itself, but more on the possibilities of the template file. For example, what options do we have when creating large(r) environments, perhaps with something like dynamic number of resources. Throughout this exploration, a simplified example will be used; creating ten S3 buckets for a demo 😉. Do you declare ten (same) definitions in the template? What if it grows to 100 buckets?

Before diving into custom resources, macros, Lambdas and such, just a short look on YAML, which, as opposed to JSON, has some built-in scripting functionality.

CloudFormation files are written in JSON. Support for YAML (v1.1) templates was introduced at a later stage to reduce the length of code and improve readability. This potentially adds some YAML processing logic.

YAML has the concept of Anchors and Aliases. Basically this copies sections (anchor) of the YAML file to defined positions (alias). This would be nice for setting a specific (set of) tags on all resources in a template, or any other repetitive section.

The example below builds on our S3 bucket list, where we need to create a number of buckets. By defining our YAML anchor on the body of the first S3 bucket definition, we alias (copy) that entire section into the other buckets. Thereby making absolutely sure that all buckets are the same as the first bucket, no copy-paste errors or stale entries, and significantly reducing the template size.

Left a full template, right a condensed template using YAML aliases and anchors

Unfortunately Anchors and Aliases are not (officially 😉) supported by CloudFormation; using a template with Anchors will result in a validation error: “Template contains errors.: Template error: YAML aliases are not allowed in CloudFormation templates”.

A workaround for this can be found by using the command-line, using the CloudFormation package command, forcing a conversion to JSON format, thereby resolving all Anchors and Aliases:

aws cloudformation package --template "`pwd`/ten_s3_buckets.yaml" --s3-bucket somebucketforcloudformation --output-template-file ten_s3_buckets.json --use-json

The processing of YAML into JSON is not AWS specific. We could do the YAML processing and conversion to JSON with any other program which supports YAML fully. For example, with Python:

cat ten_s3_buckets.yaml | python -c 'import json, sys, yaml ; json.dump(yaml.safe_load(sys.stdin.read()), sys.stdout, indent=4)' > ten_s3_buckets.json

This quick-and-dirty Python snippet processes the YAML file appropriately and generates a JSON file. (You might need to install the Python YAML library for this to work: pip install pyyaml)

Too bad YAML Anchors are not natively supported, as it would help in reducing larger templates in size and prevent other (human) errors which are accompanied by code duplication. Then again you can use any YAML processing engine to compile your template with YAML Anchors into a basic JSON template file.

Spoiler alert: Using something like Python to generate a CloudFormation template seems like something with much more potential than only YAML processing 😉. After looking at CloudFormation, or AWS native solutions, we will look at Python based solutions 😄.

Next time we will have a look at custom resources and provide you with a simple, yet fully working example.