Develop Infrastructure as Code with CloudFormation

Yunus Kılıç
Dec 30, 2019 · 6 min read

The software development industry changes very rapidly. After DevOps term becomes very popular, only writing code is not enough anymore. You should automate the software development process as can as possible. In this post, I will describe how to define our infrastructure by just writing code.

“Infrastructure as code is the process of managing and provisioning computer data centers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools”.[1]

Why do we need infrastructure as code?

  • After moving a containerized world, infrastructure can be changed or delete/create more often.
  • Also, keeping track of infrastructure setting becomes very hard, because of tons of different settings. So keeping these settings inside code, helps to figure out huge complexity.

In this post, I will use CloudFormation to define infrastructure.

“AWS CloudFormation provides a common language for you to model and provision AWS and third party application resources in your cloud environment. AWS CloudFormation allows you to use programming languages or a simple text file to model and provision, in an automated and secure manner, all the resources needed for your applications across all regions and accounts. This gives you a single source of truth for your AWS and third party resources.”[2]

I drew an architecture diagram below, to demonstrate the final.

Let’s define the components which exists inside the diagram.

Region: are separate geographic areas that AWS uses to house its infrastructure. These are distributed around the world so that customers can choose a region closest to them in order to host their cloud infrastructure there.[3]

Availability Zone: is the logical building block that makes up an AWS Region. There are currently 69 AZs, which are isolated locations — data centers — within a region.[3]

VPC: enables you to launch AWS resources into a virtual network that you’ve defined. This virtual network closely resembles a traditional network that you’d operate in your own data center, with the benefits of using the scalable infrastructure of AWS.[4]

Subnet: is “part of the network”, in other words, part of entire availability zone. Each subnet must reside entirely within one Availability Zone and cannot span zones.[5]

For simplicity, I will create 1 VPC, 2 public subnets, 2 private subnets. Subnets stand different availability zones.

PS: Deploying code commands

$sam package --template-file template.yaml --s3-bucket BUCKETNAME --output-template-file packaged.yaml
$sam deploy --template-file FILEPATH/packaged.yaml --stack-name test --capabilities CAPABILITY_IAM

First of all, let’s define parameters like below to use later.

Parameters:
EnvironmentName:
Description: An environment name that is prefixed to resource names
Type: String
Default: Test

VpcCIDR:
Description: Please enter the IP range (CIDR notation) for this VPC
Type: String
Default: 10.0.0.0/16

PublicSubnet1CIDR:
Description: Please enter the IP range (CIDR notation) for the public subnet in the first Availability Zone
Type: String
Default: 10.0.1.0/24

PublicSubnet2CIDR:
Description: Please enter the IP range (CIDR notation) for the public subnet in the second Availability Zone
Type: String
Default: 10.0.2.0/24

PrivateSubnet1CIDR:
Description: Please enter the IP range (CIDR notation) for the private subnet in the first Availability Zone
Type: String
Default: 10.0.3.0/24

PrivateSubnet2CIDR:
Description: Please enter the IP range (CIDR notation) for the private subnet in the second Availability Zone
Type: String
Default: 10.0.4.0/24

PS: 10.0.0.0/16 : Each number represents 8 digit and the first 16 digits (first two numbers) stays the same. The remaining numbers can be changed. So 256*256 Ip can be assigned.

CREATE VPC

VPC:
Type: AWS::EC2::VPC
Properties:
CidrBlock: !Ref VpcCIDR
EnableDnsSupport: true
EnableDnsHostnames: true
Tags:
- Key: Name
Value: !Ref EnvironmentName

Let’s connect this VPC with the internet. Internet Gateway is used to access internet from VPC.

InternetGateway:
Type: AWS::EC2::InternetGateway
Properties:
Tags:
- Key: Name
Value: !Ref EnvironmentName

InternetGatewayAttachment:
Type: AWS::EC2::VPCGatewayAttachment
Properties:
InternetGatewayId: !Ref InternetGateway
VpcId: !Ref VPC

CREATING SUBNETS

PublicSubnet1:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref VPC
AvailabilityZone: !Select [ 0, !GetAZs '' ]
CidrBlock: !Ref PublicSubnet1CIDR
MapPublicIpOnLaunch: true
Tags:
- Key: Name
Value: !Sub ${EnvironmentName} Public Subnet (AZ1)

PublicSubnet2:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref VPC
AvailabilityZone: !Select [ 1, !GetAZs '' ]
CidrBlock: !Ref PublicSubnet2CIDR
MapPublicIpOnLaunch: true
Tags:
- Key: Name
Value: !Sub ${EnvironmentName} Public Subnet (AZ2)

PrivateSubnet1:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref VPC
AvailabilityZone: !Select [ 0, !GetAZs '' ]
CidrBlock: !Ref PrivateSubnet1CIDR
MapPublicIpOnLaunch: false
Tags:
- Key: Name
Value: !Sub ${EnvironmentName} Private Subnet (AZ1)

PrivateSubnet2:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref VPC
AvailabilityZone: !Select [ 1, !GetAZs '' ]
CidrBlock: !Ref PrivateSubnet2CIDR
MapPublicIpOnLaunch: false
Tags:
- Key: Name
Value: !Sub ${EnvironmentName} Private Subnet (AZ2)

PublicSubnet1 stands on the first of the Availability Zone.

PublicSubnet2 stands on second.

PrivateSubnet1 stands on the first of the Availability Zone.

PrivateSubnet2 stands on second.

Public subnets should be connected to the internet. In order to handle this, we need a routing table.

PublicRouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref VPC
Tags:
- Key: Name
Value: !Sub ${EnvironmentName} Public Routes

DefaultPublicRoute:
Type: AWS::EC2::Route
DependsOn: InternetGatewayAttachment
Properties:
RouteTableId: !Ref PublicRouteTable
DestinationCidrBlock: 0.0.0.0/0
GatewayId: !Ref InternetGateway

PublicSubnet1RouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
RouteTableId: !Ref PublicRouteTable
SubnetId: !Ref PublicSubnet1

PublicSubnet2RouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
RouteTableId: !Ref PublicRouteTable
SubnetId: !Ref PublicSubnet2

Any request with related public subnet routed to the internet gateway.

PS: If you want to do the same thing for the private subnets, you should NAT Gateway.

CREATING ELASTICACHE INSTANCE WITH CODE

ServerlessSecurityGroup:
DependsOn: VPC
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: SecurityGroup for Serverless Functions
VpcId:
Ref: VPC

ServerlessStorageSecurityGroup:
DependsOn: VPC
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Ingress for Redis Cluster
VpcId:
Ref: VPC
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: '6379'
ToPort: '6379'
SourceSecurityGroupId:
Ref: ServerlessSecurityGroup

ServerlessCacheSubnetGroup:
Type: AWS::ElastiCache::SubnetGroup
Properties:
Description: "Cache Subnet Group"
SubnetIds:
- Ref: PrivateSubnet1

ElasticCacheCluster:
DependsOn: ServerlessStorageSecurityGroup
Type: AWS::ElastiCache::CacheCluster
Properties:
AutoMinorVersionUpgrade: true
Engine: redis
CacheNodeType: cache.t2.micro
NumCacheNodes: 1
VpcSecurityGroupIds:
- "Fn::GetAtt": ServerlessStorageSecurityGroup.GroupId
CacheSubnetGroupName:
Ref: ServerlessCacheSubnetGroup

Due to security reasons, I put Elasticache instance inside a private subnet. So we need to create a security group to allow 6379 port to connect Redis.

CONNECT REDIS FROM LAMBDA FUNCTION

CacheClientFunction:
Type: AWS::Serverless::Function
Properties:
Tracing: Active
CodeUri: bin/cacheClient
Handler: cacheClient
Runtime: go1.x
Role: !GetAtt RootRole.Arn
VpcConfig:
SecurityGroupIds:
- Ref: ServerlessSecurityGroup
SubnetIds:
- Ref: PublicSubnet1
Environment:
Variables:
redis_url: !GetAtt ElasticCacheCluster.RedisEndpoint.Address
redis_port: !GetAtt ElasticCacheCluster.RedisEndpoint.Port

Lambda function stands on the public subnet. We can define redis_url and redis_port according to the created ElasticCache by our definition. While writing code, we can use these environment variables to connect Redis.

!!!Important!!!

Lambda function requires some role and policy staff. Otherwise creating stack will be got an error. Let’s create a role and policy with code.

SampleManagedPolicy:
Type: AWS::IAM::ManagedPolicy
Properties:
PolicyDocument:
Version: '2012-10-17'
Statement:
- Sid: AllowAllUsersToListAccounts
Effect: Allow
Action:
- ec2:CreateNetworkInterface
- ec2:DescribeNetworkInterfaces
- ec2:DeleteNetworkInterface
- xray:PutTraceSegments
Resource: "*"

RootRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- 'sts:AssumeRole'
Path: /
ManagedPolicyArns:
- !Ref SampleManagedPolicy

The above policy will be created with minimum access.

Cache Client code:

package main

import (
"context"
"fmt"
"github.com/aws/aws-lambda-go/lambda"
"github.com/go-redis/redis"
"os"
)

func HandleRequest(ctx context.Context) (string, error) {

redisUrl := os.Getenv("redis_url")
redisPort := os.Getenv("redis_port")
client := redis.NewClient(&redis.Options{
Addr: fmt.Sprintf("%s:%s", redisUrl, redisPort),
Password: "", // no password set
DB: 0, // use default DB
})

client.Set("1", "1", 0)

return client.Get("1").String(), nil
}

func main() {
lambda.Start(HandleRequest)
}

CREATING DATABASE INSTANCE WITH CODE

ServerlessDBSecurityGroup:
DependsOn: VPC
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Ingress for Redis Cluster
VpcId:
Ref: VPC
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: '5432'
ToPort: '5432'
SourceSecurityGroupId:
Ref: ServerlessSecurityGroup

ServerlessDBSubnetGroup:
DependsOn: ServerlessDBSecurityGroup
Type: AWS::RDS::DBSubnetGroup
Properties:
DBSubnetGroupDescription: "DB Subnet Group"
SubnetIds:
- Ref: PrivateSubnet1
- Ref: PrivateSubnet2

PostgresqlInstance:
DependsOn: VPC
Type: AWS::RDS::DBInstance
Properties:
AllocatedStorage: 30
DBInstanceClass: db.t2.micro
DBName: postgres
Engine: postgres
MasterUsername: CacheClient
MasterUserPassword: ChangeIt2
DBSubnetGroupName: !Ref ServerlessDBSubnetGroup
VPCSecurityGroups:
- "Fn::GetAtt": ServerlessDBSecurityGroup.GroupId
DbClientFunction:
Type: AWS::Serverless::Function
Properties:
Tracing: Active
CodeUri: bin/dbClient
Handler: dbClient
Runtime: go1.x
Role: !GetAtt RootRole.Arn
VpcConfig:
SecurityGroupIds:
- Ref: ServerlessSecurityGroup
SubnetIds:
- Ref: PublicSubnet1
Environment:
Variables:
db_url: !GetAtt PostgresqlInstance.Endpoint.Address
db_port: !GetAtt PostgresqlInstance.Endpoint.Port

Creating DB is very similar to the cache. The important point is your DB instance should stand on at least two availability zone.

DB client code:

package main

import (
"context"
"fmt"
"github.com/aws/aws-lambda-go/lambda"
"github.com/jinzhu/gorm"
_ "github.com/jinzhu/gorm/dialects/postgres"
"os"
)

type MyEvent struct {
Name string `json:"name"`
}

func HandleRequest(ctx context.Context, name MyEvent) (string, error) {

dbUrl := os.Getenv("db_url")
dbURI := fmt.Sprintf("host=%s user=CacheClient dbname=postgres sslmode=disable password=ChangeIt2", dbUrl)
fmt.Println(dbURI)
db, err := gorm.Open("postgres", dbURI)
if err != nil {
return "err", err
}

db.AutoMigrate(&Entity{})
var ent = &Entity{}
ent.Text = name.Name
db.Save(&ent)

return fmt.Sprint(&ent.ID), nil
}

func main() {
lambda.Start(HandleRequest)
}
package main

import "github.com/jinzhu/gorm"

type Entity struct {
gorm.Model
Text string
}

All is done. Now we created network, cache instance, DB instance with code. We bind the output of them with our serverless functions. Cache and DB clients used created architecture by code.

The full version of template.yaml:

https://github.com/yunuskilicdev/infrastructureascode

Cites:

1-) https://stackify.com/what-is-infrastructure-as-code-how-it-works-best-practices-tutorials/

2-) https://aws.amazon.com/cloudformation/

3-)https://cloudacademy.com/blog/aws-regions-and-availability-zones-the-simplest-explanation-you-will-ever-find-around/

4-)https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html

5-)https://www.infoq.com/articles/aws-vpc-explained/

Analytics Vidhya

Sign up for Analytics Vidhya News Bytes

By Analytics Vidhya

Latest news from Analytics Vidhya on our Hackathons and some of our best articles! Take a look.

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

Analytics Vidhya

Analytics Vidhya is a community of Analytics and Data Science professionals. We are building the next-gen data science ecosystem https://www.analyticsvidhya.com

Yunus Kılıç

Written by

I have 7 years of experience in high-quality software application development, implementation, and integration.

Analytics Vidhya

Analytics Vidhya is a community of Analytics and Data Science professionals. We are building the next-gen data science ecosystem https://www.analyticsvidhya.com

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store