Merapar DevOps — CloudFormation macros to create dynamic CloudWatch Dashboards
In a series of blog posts we will focus on some of the best practices we use within Merapar to evolve our DevOps practices we have built around the AWS platform. Why? Because we think it’s fun to share knowledge and to learn from others in the industry.
Situation
In this blog post we will focus on how we set up monitoring using a single AWS CloudWatch dashboard for a project where we have multiple AWS CloudFront Distributions.
The metrics that need to be monitored will be the same for all of the Distributions as the type of traffic is very similar. As the number of Distributions will change often we need a flexible solution to prevent repetitive tasks of adding / removing references to the Distribution in our CloudFormation templates. We also have a strong need to have the same code base between all our DTAP environments.
Requirements:
- Creation of AWS CloudWatch dashboard must be fully automated
- Multiple CloudFront Distributions within a single dashboard
- No code changes required on CloudFormation dashboard to add new Distributions
Solution
The solution we have chosen uses a CloudFormation template and the macro functionality to perform the required transformations to the template.
What are CloudFormation macros
CloudFormation macros are simply AWS Lambda functions that you create and can be used to transform your input CloudFormation template. This gives you the ability to do search and replace operations or to transform the template completely.
Macros are run after uploading the template and before applying it. It is possible to see both the input and the processed template within the AWS CloudFormation console to make it easier to debug.
It is possible to have a macro process the complete template using the Transform section
Or to limit the scope to a smaller part of the template using Fn::Transform
For more specific information about AWS CloudFormation macros take a look at this user guide from AWS.
CloudFormation template
For our solution we have chosen to use the smaller scoped solution and we created to following template:
The Fn::Transform is used to scope just the dashboard and the Distribution variable is passed to the transform. We have used the ###xxx### markers to easily find and replace the values we need to transform.
In our setup there is a deployment script that will first deploy all required CloudFront Distributions based on a configuration file and once completed, it will deploy the dashboard with that same list. This list of Distributions is passed as the DistributionConfiguration parameter with a JSON object as input for this template.
Macro Lambda code
The ExpandDashboard macro is a Python3 AWS Lambda that is deployed separately from the CloudFront Distributions and the CloudWatch dashboard.
This Lambda function will loop through the input JSON object and add the configuration required for the 4xx, 5xx errors and request widgets for all distributions.
Dashboard
This is what the dashboard will look like:
