Serverless Stack — CI/CD — Blue-Green Deployments

Check out this awesome Serverless Stack with built-in CI/CD and Blue-Green Deployments

Randy Findley
Jan 9, 2018 · 7 min read

I love building CloudFormation stacks, crazy I know… I also love serverless event-driven architectures, who doesn’t…

I wanted to create a reusable stack that I could easily use to build web applications.

The stack consists of an API, UI, and Async Tasks. This isn’t the cool part.

The cool part is the built-in CI/CD via CodePipeline and Blue-Green deployments of Lambda.

CI/CD and Blue-Green deployments are very important for the stability and health of an application. It is important to deliver often and fail fast.

Code

The code. Check it out, run it, and let me know what you think…

Architecture

Lets take a look at the architecture before we talk about the CI/CD and Blue-Green deployments.

CI/CD

When you push code or infrastructure changes to this application CodePipeline builds and updates everything. Your infrastructure stacks are updated. Your code is built, tested, and deployed.

You can make changes and push all day long. Development will be fast and predictable. Happy days.

Oh, and there is a sandbox and a prod environment, so you can test out your changes in sandbox prior to pushing them to prod. It’s nice to be able to test changes in a sandbox environment with your team before you go live with them.

Here is what the pipeline looks like:

Lets take a closer look at each pipeline stage and action. A pipeline consists of stages. Each stage consists of actions.

How you organize your stages and actions depends on 2 things. The order you want your stages and actions to be processed in, and the stage input and output dependencies.

Stages can be organized in series or in parallel or both. This is defined by the stage input and output artifacts.

Actions can also be organized in series or in parallel or both. This is defined by the action order property and the input and output artifacts.

If the input artifact for each stage is the output artifact for the previous stage then they will be in series. If the input artifact for a stage is the output artifact of several previous stages, and those previous stages don’t depend on each other, then those previous stages will be in parallel.

Enough about CodePipeline, lets learn more about our pipeline:

Source Stage — GitHub Push

The source stage is triggered on every GitHub push.

- Name: Source
Actions:
- Name: CloneRepository
ActionTypeId:
Category:
Source
Owner: ThirdParty
Version: 1
Provider: GitHub
OutputArtifacts:
- Name: GitSource
Configuration:
Owner:
!Ref GitHubOwner
Branch: 'master'
Repo: !Ref GitHubRepo
OAuthToken: !Ref GitHubToken
RunOrder: 1

The prod pipeline uses the master GitHub branch and the sandbox pipeline uses the develop GitHub branch.

Tasks Package Stage

This stage uses CodeBuild to build & test the Lambda, then package the CloudFormation template.

- Name: TasksPackage
Actions:
- Name: Package
ActionTypeId:
Category:
Build
Owner: AWS
Provider: CodeBuild
Version: 1
Configuration:
ProjectName:
!Ref TasksCodeBuildProd
InputArtifacts:
- Name: GitSource
OutputArtifacts:
- Name: TasksOutput
RunOrder: 1

Our Lambda within the CloudFormation template uses a relative CodeUri property. When using the CodeUri property you have to use CloudFormation package(aws cloudformation package). CloudFormation package does 2 things. First it uploads your Lambda at the CodeUri location as a zip file to S3, then it exports the template with the CodeUri replaced with the zip file location in S3.

Here is the entire CodeBuild buildspec.yml.

version: 0.2
phases:
install:
commands:
pre_build:
commands:
- echo Installing source NPM dependencies...
- cd ./stacks/tasks && npm install
build:
commands:
- echo Testing the code
- npm test
- echo Removing dev dependencies
- rm -Rf node_modules
- npm install --production
post_build:
commands:
- aws cloudformation package --template-file tasks.stack.yml --s3-bucket ${Bucket} --output-template-file tasks.stack.output.yml
artifacts:
base-directory:
'stacks/tasks'
type: zip
files:
- tasks.stack.output.yml

This CodeBuild exports the packaged CloudFormation template. This will be used in the next stage.

Tasks Stack Stage

This stage mutates (create/update) the tasks stack.

- Name: TasksStack
Actions:
- Name: CreateChangeSet
InputArtifacts:
- Name: TasksOutput
ActionTypeId:
Category:
Deploy
Owner: AWS
Version: 1
Provider: CloudFormation
Configuration:
TemplatePath:
"TasksOutput::tasks.stack.output.yml"
ActionMode: CHANGE_SET_REPLACE
Capabilities: CAPABILITY_NAMED_IAM
RoleArn: !GetAtt CloudFormationRole.Arn
StackName: !Sub "${AWS::StackName}-tasks-prod"
ChangeSetName: !Sub "${AWS::StackName}-tasks-prod-cs"
RunOrder: 1
- Name: ExecuteChangeSet
InputArtifacts:
- Name: TasksOutput
ActionTypeId:
Category:
Deploy
Owner: AWS
Version: 1
Provider: CloudFormation
Configuration:
ActionMode:
CHANGE_SET_EXECUTE
Capabilities: CAPABILITY_NAMED_IAM
RoleArn: !GetAtt CloudFormationRole.Arn
StackName: !Sub "${AWS::StackName}-tasks-prod"
ChangeSetName: !Sub "${AWS::StackName}-tasks-prod-cs"
RunOrder: 2

The stage has two actions.

The first action creates a CloudFormation ChangeSet. You can see that the input artifact is the output artifact of the Tasks Package stage. Take a look at the TemplatePath, it is from the CodeBuild package command.

The second action executes the CloudFormation ChangeSet mutating the stack.

API Package & API Stack Stages

These stages are exactly the same as the Tasks stages. They package the Lambda & CloudFormation template, then create the CloudFormation ChangeSet, and finally mutate that ChangeSet.

UI Stack Stage

The UI stack stage does things in the reverse order. Instead of building, testing, and uploading the code, then mutating the stack. It mutates the stack and then builds, tests, and uploads the code.

- Name: UI
Actions:
- Name: Stack
InputArtifacts:
- Name: GitSource
ActionTypeId:
Category:
Deploy
Owner: AWS
Version: 1
Provider: CloudFormation
Configuration:
TemplatePath:
"GitSource::stacks/ui/ui.stack.yml"
ActionMode: CREATE_UPDATE
Capabilities: CAPABILITY_NAMED_IAM
RoleArn: !GetAtt CloudFormationRole.Arn
StackName: !Sub "${AWS::StackName}-ui-prod"
ParameterOverrides: !Sub |
{
"Domain": "${Domain}",
"TLD" : "${TLD}"
}
RunOrder: 1
- Name: Deploy
ActionTypeId:
Category:
Build
Owner: AWS
Provider: CodeBuild
Version: 1
Configuration:
ProjectName:
!Ref UICodeBuildProd
InputArtifacts:
- Name: GitSource
RunOrder: 2

The first action mutates the CloudFormation stack and creates the S3 bucket we need in the next action.

The second action executes a CodeBuild that uploads the code to the S3 bucket thereby deploying the static website.

Here is the CodeBuild buildspec.yml.

version: 0.1
phases:
install:
commands:
pre_build:
commands:
build:
commands:
- aws s3 sync stacks/ui/www "s3://$(aws cloudformation describe-stacks --stack-name ${StackName} --query "Stacks[0].Outputs[0].OutputValue" --output text)" --acl bucket-owner-full-control --acl public-read --delete --cache-control "max-age=1" --exclude stacks/ui/www/assets
- aws s3 sync stacks/ui/www/assets "s3://$(aws cloudformation describe-stacks --stack-name ${StackName} --query "Stacks[0].Outputs[0].OutputValue" --output text)/assets" --acl bucket-owner-full-control --acl public-read --delete --cache-control "max-age=31536000"
post_build:
commands:

This is interesting. We get the S3 bucket name from the previous step by fetching the output parameter from the stack. Here is the command:

$(
aws cloudformation describe-stacks
--stack-name ${StackName}
--query "Stacks[0].Outputs[0].OutputValue"
--output text
)

That’s it. That’s how the pipeline performs CI/CD for our infrastructure and code.

Now lets take a look at the blue-green deployments.

Blue-Green Deployments

Within our api CloudFormation template we have a Lambda.

LambdaFunction:
Type:
AWS::Serverless::Function
Properties:
Handler:
index.handler
Timeout: 5
Role: !GetAtt IamRoleLambdaExecution.Arn
CodeUri: ./
Runtime: nodejs6.10
AutoPublishAlias: live
DeploymentPreference:
Type:
Canary10Percent5Minutes
Alarms:
- !Ref 5xxAlarm
- !Ref 4xxAlarm
- !Ref LatencyAlarm
Environment:
Variables:
TasksSnsTopic:
Fn::ImportValue:
!Sub "${TasksStack}-SNSTopic"

This Lambda uses the Serverless Application Model (SAM), which is a CloudFormation transformer.

When using SAM types within a CloudFormation template you need to add the transform definition. Transform: AWS::Serverless-2016–10–31.

I’m not sure why SAM exists at all. Why wasn’t CloudFormation extended to include these new SAM features? I read somewhere that SAM is less verbose. Is that the only reason? If so that doesn’t make up for the fragmentation and confusion SAM brings. Just my 2 cents…

Take a look at the DeploymentPreference property. This is where we define the Safe Traffic Shifting.

We use Canary10Percent5Minutes which routes 10% of traffic to the new Lambda, then waits 5 minutes, if all is good (no alarms go off) the remaining traffic is routed and the deployment is done. There are 3 types of traffic shifting.

  • LinearXPercentYMinutes: Traffic to new version will linearly increase in steps of X percentage every Y minutes. Ex: Linear10PercentEvery10Minutes will add 10 percentage of traffic every 10 minute to complete in 100 minutes.
  • CanaryXPercentYMinutes: X percent of traffic will be routed to new Version once, and wait for Y minutes in this state before sending 100 percent of traffic to new version. Some people call this as Blue/Green deployment. Ex: Canary10Percent15Minutes will send 10 percent traffic to new version and 15 minutes later complete deployment by sending all traffic to new version.
  • AllAtOnce: This is an instant shifting of 100% of traffic to new version. This is useful if you want to run run pre/post hooks but don’t want a gradual deployment. If you have a pipeline, you can set Beta/Gamma stages to deploy instantly because the speed of deployments matter more than safety here.

The alarms property is where you define CloudWatch alarms to monitor while your traffic is being shifted. If any of these alarms go off the deployment will be reverted.

Alarms:
- !Ref 5xxAlarm
- !Ref 4xxAlarm
- !Ref LatencyAlarm

Our alarms are based on the API Gateway traffic. Are we getting too many 5xx or 4xx errors, or did the latency spike? If so something is wrong with our new version and we don’t want to deploy it.

Here are the alarms.

5xxAlarm:
Type:
AWS::CloudWatch::Alarm
DependsOn: RestApi
Properties:
AlarmDescription:
5xx alarm for api gateway
Namespace: 'AWS/ApiGateway'
MetricName: 5XXError
Dimensions:
- Name: ApiName
Value: !Ref RestApi
Statistic: Sum
Period: '60'
EvaluationPeriods: '3'
Threshold: '10'
ComparisonOperator: GreaterThanOrEqualToThreshold
4xxAlarm:
Type:
AWS::CloudWatch::Alarm
DependsOn: RestApi
Properties:
AlarmDescription:
4xx alarm for api gateway
Namespace: 'AWS/ApiGateway'
MetricName: 4XXError
Dimensions:
- Name: ApiName
Value: !Ref RestApi
Statistic: Sum
Period: '60'
EvaluationPeriods: '3'
Threshold: '10'
ComparisonOperator: GreaterThanOrEqualToThreshold
LatencyAlarm:
Type:
AWS::CloudWatch::Alarm
DependsOn: RestApi
Properties:
AlarmDescription:
latency alarm for api gateway
Namespace: 'AWS/ApiGateway'
MetricName: Latency
Dimensions:
- Name: ApiName
Value: !Ref RestApi
Statistic: Average
Period: '60'
EvaluationPeriods: '3'
Threshold: '25000'
ComparisonOperator: GreaterThanOrEqualToThreshold

That’s it. I hope you enjoyed this post and get some use out of this stack.

Please clap if you liked this article.

HackerNoon.com

how hackers start their afternoons.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade