Macros: DRY has come to your CloudFormation templates.

When you manage a whole infrastructure as code, as small as it can be, you sometimes have to do copy-paste of the code to get things done. Other times, you have to do manipulation of your templates before they go live. The problem is, if you have to manipulate them every time you send them in your infrastructure, you lose time and also insert a human error factor.

That’s where the macros get in the ring.

One of the problems on the table was that we had ECS containers with redundant environment configuration and, with those duplicates, the template was exploding the limit of 51200 bytes. After a bit of research and some fiddling, the macros seemed to be our best bet.

A macro is a lambda function that receives 7 parameters and must return 3 outputs to be successful. The most important parts of the inputs are the TransformId, the id provided by CloudFormation to uniquely identify the macro execution, and the Fragment, which is a JSON version of your template, regardless of its original format. We apply our modifications to the Fragment and return it with the TransformId and the Status, and voila! It sounds easy when presented that way. Well, it is! And here’s a quick example :

A quick macro that changes a stack description

First things first, that macro needs an environment to live. We’ll use Serverless Framework to deploy a CloudFormation stack with those 2 resources: a lambda with the handler pointing to our python script and the macro definition. No outputs, no IAM role, nothing. Here’s an example of the most basic Serverless template for our example macro :

serverless.yml bare minimum file to make our macro live

Of course, this baby will not work on its own, as we did not provide it the trigger in our template, but the deployment part is now done and we won’t touch it again. In our example, the macro is not really flexible. If it is not called with a property named Description at the root level of the fragment, it will crash. Keep in mind that there are two ways to call a macro:

Either by calling it in the Transform section of the template, thus giving it access to the whole template :

The macro call will only change the template description, not every occurrence

Or it can be called by using the Fn::Transform, but it will only have access to the scope where it is called :

The macro will only have access to it’s scope where it was called

In our case, there was a bit of thinking needed in the code of the macros as to how we will trigger it and how will this trigger work. We wanted it to work no matter where the trigger was located in the template and needed it to work with no additional input. The simplest solution was to include its trigger directly in the template with the block to copy as its value. This way, we search for the trigger in the fragment and we use its value as to where we get the content to replace.

Here is the code that is used in the example. Note that we use the dpath library to navigate through the JSON template :

In the code above, we first search for every occurrences of the trigger (inheritFrom) in the environment section of the template. Once that is done, we extract its value and then fetch the content where the value was referring. We then delete the trigger to prevent infinite loop and replace the content while overriding with the value specified, if any.

Once this is done, the next to last step is to add in our template that we want a macro to modify it. Since in our case we want it to run through the whole template and find its trigger alone, we configure it in the root section of the template called Transform. We add a list parameter with the name of the macro defined in our Serverless template and boom! We now have a template that is almost half the size it originally was since we reduced redundancy.

Here is an example of template in which the macro will copy the environment section of the source into the destination, while overriding the value of PROPERTY1 with 4:

The final step is, of course, to deploy the stack with the macro and then, every time we will have an update or a modification to our stack with the template containing the macro call, it will run before anything in it. The only caveat is that if you change a parameter, or anything that doesn’t trigger a template update, the macro will not execute.

So, with as little as a 34 lines of Python code and a minimalist Serverless template, we saved around 20 000 bytes in the template. Also, we nullify the risk of human error by removing the need to copy the templates modifications in every section where it appears.

Alternatives to consider :

• aws-cdk : TypeScript library to generate and deploy templates (In development)
• troposphere : Python library to generate templates