AWS DevOps with CloudFormation

CloudFormation is a tool from AWS for automating infrastructure provisioning in AWS using the infrastructure-as-code model. The tool is free as in, all the CloudFormation functionalities are free to use, but the user will be charged for all the AWS resources that will be created through CloudFormation. Main advantage is that, being an AWS product, you can always expect up-to-date compatibility with AWS resources.

Templates

Developers will model their AWS infrastructure using JSON or YAML syntax in Cloud Formation Template files. File extension could be anything you want.

Each Template could have the below sections in it. Resources is the only mandatory section.

• AWSTemplateFormatVersion:“2010–09–09” — version of the CloudFormation template. Only accepted value is ‘2010–09–09’
• Description: Enables you to include arbitrary comments about your template. (Optional) eg: “This template will create an EC2”
• Mappings: Collection of Key-Value pairs which can be used to set values. (Optional) eg: List of AMI Ids for each AWS Region
• Resources: Declares the AWS resources that you want to include in the stack. (Required) eg: an EC2 instance, a security group
• Parameters: Parameters enable you to input custom values to your template each time you create or update a stack. (Optional) eg: SSH Key for logging into the EC2.
• Outputs: Declares output values that you can import into other stacks, return in response, or view on the AWS CloudFormation console. (Optional) eg: IP address of the EC2 instance created.

Each template could have one or more resources declared in it. They could have dependancies on each other, and cloudformation will make sure to create the independent resource first and then create the resources dependent on it.

Stack

When you deploy some infra using cloudformation from a template, a stack is created. Think of it like you first create a docker image, and when you run the image using docker run, you create a container. Once you have a stack from a template, all the resources in the stack are tracked together. If the cloudformation is successful, everything in stack is created. If even one resource fails, everything fails (anything already created will be rolled back)

CF Demo 1

Now lets create an EC2 instance using CloudFormation.

Standard format in the Cloud Formation template is given below.

AWSTemplateFormatVersion: "2010-09-09"
Description: "Description"
Resources:
  LogicalIDForTheResource:
    Type:
    Properties:

The complete code is:

AWSTemplateFormatVersion: "2010-09-09"
Description: Create EC2 with CF
Resources:
  TestInstance: #Create a resource with the ID TestInstance
    Type: AWS::EC2::Instance # Resource is of type EC2 instance
    Properties: # Resource Properties given below
      ImageId: ami-090fa75af13c156b4 #AMI ID for EC2(Amazon Linux2)
      InstanceType: t2.micro #EC2 instance type in AWS

Lets save this as ec2.yaml (Cloudformation does not care about the file extension, but keeping yaml as .yaml will help with any syntax highlighting and other functionalities you might have in your text editor like VS Code)

We can run the cloudformation from AWS Console or from terminal using aws CLI. Both places, make sure you are using an IAM user with the required permission for creating EC2 instances with properties as given in the template, and you have not crossed the limits in your accounts. Note: This file should be used from us-east-1 region in AWS, because the AMI ID in template exists in that region.

From Console:

CloudFormation — Create Stack — With New Resources

Template is ready — Import File. Select the file created.

Enter Stack name. We dont have any parameters in this template. Click Next.

Keep all other options default and Click Create Stack. After some time, creation is complete. Timeline of the cloud formation process is shown.

In the resources tab you can see the resources created. Physical ID shows a link to the resource in EC2 console.

Congratulations! You have created your first resource in AWS with cloudformation.

To do the same from CLI, you can run below command:

aws cloudformation create-stack --template-body file://ec2.yaml --stack-name Test-EC2

Now if you delete the stack from the cloudformation stack console, it will not just delete the entry inside cloudformation, it deletes the resources too. Remember — Stack = Deployed resources from template.

To delete from CLI:

aws cloudformation delete-stack --stack-name Test-EC2

CF Demo 2 — More features

Mappings

Note that, for this simple example, we have hard-coded the AMI ID which we confirmed before hand that is available in the region we are using (us-east-1) and we made sure to run the cloudformation in that region. In practice this may not be the case. How can we make sure that the cloudformation always picks an AMI that exists in the region, regardless of which region it runs in?

Mappings will help us here.

Mappings help to add key-value pairs in a table form in the template, and select and use a value from it using one or more keys.

A sample Mapping section might look like below:

Mappings:
  AnExampleMap:
    TopLevelKey01:
      Key01: Value01
      Key02: Value02

    TopLevelKey02:
      AnotherKey: AnExampleValue

    TopLevelKey03:
      AFinalKey: ADifferentValue

In this map, by specifying the map name (AnExampleMap) and Top level key (TopLevelKey01) and Second level key (Key01) we can derive Value01. For this we use cloudformation intrinsic functions FindInMap and Ref. FindInMap takes the map name, and the Ref takes a key as parameters.

For our purpose, lets have a map of valid AMI IDs, (we will limit it to US regions for this exercise)

Mappings:
  AWSAMIRegionMap:
    us-east-1:
      AmazonLinux2: ami-090fa75af13c156b4
    us-east-2:
      AmazonLinux2: ami-051dfed8f67f095f5
    us-west-1:
      AmazonLinux2: ami-0e4d9ed95865f3b40
    us-west-2:
      AmazonLinux2: ami-0cea098ed2ac54925

Here map name is AWSAMIRegionMap, Top level key is the AWS Region (We will fetch this from CF environment shortly) Second level key is the string AmazonLinux2.

Instead of having ImageID hardcoded, now we will fetch it using Region.

ImageId: 
        Fn::FindInMap:  #Look in a  map 
        - AWSAMIRegionMap # Map name is given
        - Ref: AWS::Region # Look for this key first 
        - AmazonLinux2 # Inside that key, look for this key

AWS::Region is a pseudo parameter which will always contain the current session AWS region. It is set by CloudFormation service. We use Ref: function to get that value.

All such parameters are documented here:

Pseudo parameters reference

All Intrinsic functions are documented here:

Intrinsic function reference

The complete code with mapping looks like below:

AWSTemplateFormatVersion: "2010-09-09"
Description: Create EC2 with CF v2.0
Resources:
  TestInstance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: 
        Fn::FindInMap:
        - AWSAMIRegionMap
        - Ref: AWS::Region
        - AmazonLinux2
      InstanceType: t2.micro
Mappings:
  AWSAMIRegionMap:
    us-east-1:
      AmazonLinux2: ami-090fa75af13c156b4
    us-east-2:
      AmazonLinux2: ami-051dfed8f67f095f5
    us-west-1:
      AmazonLinux2: ami-0e4d9ed95865f3b40
    us-west-2:
      AmazonLinux2: ami-0cea098ed2ac54925

Parameters

Now, let’s say we want to prompt the user to enter a Tag value as name (not really a production case, tagging is usually controlled and automated) We can use Parameters for that.

How do we know what all properties can be set on the AWS Resource using Cloud Formation? Here is the documentation:

AWS::EC2::Instance

We can add an Parameter to input the tag via:

Parameters:
  EC2NameTag:
    Description: Name Tag
    Type: String

This causes a prompt for the parameter during cloudformation run.

The Parameter value can be fetched with Ref: Parameter syntax and assigned to tag:

Tags: 
      - Key: Name
        Value: 
          Ref: EC2NameTag

Outputs

What if we want to print the IP address of the EC2 instance? Output will help us here. GetAtt intrinsic function can be used to access the properties of the AWS resource that was created.

Below given code will output the PublicIP attribute of the EC2 created. Note that we are using the logical ID of the resource (TestInstance) in the function.

Outputs:
  InstanceIP:
    Value:
      Fn::GetAtt:
      - TestInstance
      - PublicIp
    Description: Instance IP

Complete template is given here:

CloudFormationDemos/EC2-v2.yaml at main · manumaan/CloudFormationDemos

When we run this in us-east-1 it picks up the AMI ID that is defined for us-east-1, and when we run it in us-west-2 it picks the AMI for us-west-2. Tags supplied via the Parameters are also reflecting.

It prints the Public IP in the outputs tab:

Nested Stack

What if you had an RDS database used for different applications, and you want to keep its template and stack separate from application stacks that use it? Or you want to have modularity or code-reuse in your templates? In all these cases we can achieve those by using a nested stack. Nested stacks usually have one “root” template/stack, and it calls all the other “child” templates/stacks from that.

A Nested Stack resource would look like this in template:

Resources:
  Stack1:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: 'Path/To/Template' #Path where templates are
      Parameters: #Parameters  sent from root to child Stack1
        ExampleKey: ExampleValue  
        Type: String
  Stack2:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: 'Path/To/Template' #Path where templates are
      Parameters: #Parameters  sent from root to child Stack2
        ExampleKey: ExampleValue

Note the AWS::CloudFormation::Stack type. Instead of having one or more resources, A nested stack template will have more than one stack types defined. Each stack will point to a path where the actual template files are kept.

You might wonder, how do i send data from and to child templates? Outputs and Parameters help with this.

For eg: We want to create a Security Group and associate the EC2 to that security group. How this would work in nested group is so:

1. Root template calls Security Group template. Security Group template has an output defined:

Outputs:
  SGID:
    Value:
      Fn::GetAtt:
      - ec2sg
      - GroupId
    Description: SG Id

2. Now back in the root template we can get the value in SGID (which is security group ID) using the GetAtt syntax:

!GetAtt SGStack.Outputs.SGID

3. Note the usage of the stack name. So when we call the EC2 template from root to create EC2, we can send the security group ID:

Resources:
  EC2Stack:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: !Sub https://${S3BucketName}.s3.amazonaws.com/ec2.yaml
      TimeoutInMinutes: 20
      Parameters:
        SecurityGroupIds: !GetAtt SGStack.Outputs.SGID
        Tags: 
          Ref: EC2NameTag

To receive this parameter in the EC2 template we should have parameters defined with same name, and make sure to actually use it.

Parameters:
  SecurityGroupIds:
    Description: Security Group ID
    Type: String

Because security group ID is a single string and the parameter for Security Group IDs expects a list of strings, we use split function to create a list of one item.

Resources:
  TestInstance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: 
        Fn::FindInMap:
        - AWSAMIRegionMap
        - Ref: AWS::Region
        - AmazonLinux2
      InstanceType: t2.micro
      SecurityGroupIds: 
        Fn::Split: [",", Ref: SecurityGroupIds]

For a nested template, all templates should be inside an S3 bucket. How do we pass the S3 bucket to the templates? Using a parameter in root template.

Remember we used to have a Parameter for the Name tag of the EC2. This also becomes a parameter inside root template.

Both these can be passed with Ref function to the child templates that need it. For eg:

Resources:
  EC2Stack:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: !Sub https://${S3BucketName}.s3.amazonaws.com/ec2.yaml
      TimeoutInMinutes: 20
      Parameters:
        SecurityGroupIds: !GetAtt SGStack.Outputs.SGID
        Tags: 
          Ref: EC2NameTag

CF Demo 3 — Nested Stack

The complete code for the nested template demo is kept here:

CloudFormationDemos/demo3 at main · manumaan/CloudFormationDemos

We upload all template files to an S3 bucket.

main.yaml is the root template. Provide path to that in CloudFormation.

Enter EC2 Name Tag, Bucket Name parameter values.

Because we are creating stacks inside stacks, we need to acknowledge these capabilities. Check those.

On stack creation completion you can see on the left side one normal stack and 2 nested stacks. Events, outputs etc can be checked individually for these.

Output values: EC2 IP address

Instance created.It has the expected Name Tag and it is associated with the Security Group that was created.

For deleting the resources from a nested stack, be sure to click delete on the root stack.

AWS Cloud Formation is a big topic and we have covered the essentials in this post. You can expect one more post on this topic where I would cover some more on this area : https://awstip.com/aws-devops-with-cloudformation-2-f409bcc4e664?source=friends_link&sk=db3eaffcd1343bf2dbb9d11933defc48

Hope this was helpful!