Setting up CI/CD for Lambda Functions using AWS CodePipeline+BitBucket+CloudFormation
How to deploy AWS Lambda functions of Python using Bitbucket, CodeBuild, and CloudFormation and orchestrated by CodePipeline.
This is my Architecture diagram:
Prerequisite:
- Create a BitBucket Account and a branch
- Create a bucket for packaged source code
- Create a bucket for packaged Cloudformation template file
Permission:
- Lambda IAM Role
{
"Statement": [
{
"Action": [
"apigateway:*",
"codedeploy:*",
"lambda:*",
"cloudformation:CreateChangeSet",
"iam:GetRole",
"iam:CreateRole",
"iam:DeleteRole",
"iam:PutRolePolicy",
"iam:AttachRolePolicy",
"iam:DeleteRolePolicy",
"iam:DetachRolePolicy",
"iam:PassRole",
"s3:GetObject",
"s3:GetObjectVersion",
"s3:GetBucketVersioning"
],
"Resource": "*",
"Effect": "Allow"
}
],
"Version": "2012-10-17"
}
- CodePipeline IAM Role: Create by Console
- CodeBuild IAM Role
It will be use on Build stage setting.
Please create and attach this policy to your codebuild role after you created the Codebuild role on console.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ListObjectsInBucket",
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::your-default-artifact-store-bucket",
"arn:aws:s3:::your-sourcecode-store-bucket"]
},
{
"Sid": "AllObjectActions",
"Effect": "Allow",
"Action": "s3:*Object",
"Resource": [
"arn:aws:s3:::your-artifact-store-bucket/*"
"arn:aws:s3:::your-sourcecode-store-bucket/* ]
}
]
}
- CloudFormation IAM Role
It will be use on Deploy stage for CloudFormation.
{
"Statement": [
{
"Action": [
"codedeploy:*",
"lambda:*",
"cloudformation:CreateChangeSet",
"iam:GetRole",
"iam:CreateRole",
"iam:DeleteRole",
"iam:PutRolePolicy",
"iam:AttachRolePolicy",
"iam:DeleteRolePolicy",
"iam:DetachRolePolicy",
"iam:PassRole",
"s3:GetObject",
"s3:GetObjectVersion",
"s3:GetBucketVersioning"
],
"Resource": "*",
"Effect": "Allow"
}
],
"Version": "2012-10-17"
}
Steps:
- CLONE repository and PUSH source code, build template, and deploy template files to the repository
- Create pipeline in Codepipeline
- Create Codebuild project
- Source code is deployed by CodePipeline automatically
1. CLONE repository and PUSH source code, build template, and deploy template files to the repository
My file is as below in BitBucket:
- buildspec.yml
1.This is the yml to tell Codebuild how to build it.
2. install: define runtime version
3. build: install some external library
4. post_build: The output file would be in the artifact zip that is passed by CodePipeline from CodeBuild to CloudFormation action.
version: 0.2phases:
install:
runtime-versions:
python: 3.7
build:
commands:
- echo "install pandas and numpy"
- mkdir dep
- pip install pandas -t ./dep
- pip install numpy -t ./dep
post_build:
commands:
- aws cloudformation package --template-file function.yml --s3-bucket $S3BUCKET --force-upload --output-template-file function-out.ymlartifacts:
type: zip
files:
- function.yml
- function-out.yml #Use this template for Deploy stage
- template.yml
I use a SAM template for CodeBuild here. You should define the value as below,Codebuild will compile source code:
- CodeUri: your code path
- FunctionName: Lambda Function Name
- Handler: Your lambda handler
Resources:
SampleLambdaFunction:
Properties:
CodeUri: . #Directory the source file is
FunctionName: cicd-lambda-print_log
Handler: lambda_function.lambda_handler
Runtime: python3.7
Type: AWS::Serverless::Function
Transform: AWS::Serverless-2016-10-31
- lambda_function.py
My Lambda Function is as below:
import numpy as np
import pandas as pddef lambda_handler(event, context):
return 'commit from 20201222 14:03'
And we can push these three file to the repo by using the cmd on the local repo.
git add .
git commit -m "first commit"
git push
2.Create pipeline in Codepipeline
Click “Create Pipeline”
Choose pipeline settings
Pipeline Name- Testpipeline
Service role- New Service role and check
Advance settings- Custom location: Specified a S3 bucket for packaged Cloudformation template file. The output file would be in the artifact zip that is passed by CodePipeline from CodeBuild to CloudFormation action.
Add source stage
Select “Bitbucket” and “connect to Bitbucket”
Create a connection to Bitbucket
Input Connection Name and connect Bitbucket
Grant access to AWS codestar
1.Click “Install a new app”
2.Connect
Authorize for codestar to read info as below
Connection will be input in some value and Click Connect
Go to Bitbucket>Personal Setting to check the link between codestar and Bitbucket
Add source stage
Select your repo and Branch on Bitbucket
Add Build stage
Build provider -AWS CodeBuild
Region-Your region
Project name- Create project and move to another page as below
Create build project
Project Name- Your project name
Environment image- Managed image
Operating system- Ubuntu
Runtimes- Standard
Image- aws/codebuild/standard:3.0
Environment type- Linux
Service Role- check
Environment variables- Input “S3BUCKET” (The variables that you specified in the buildspec.yml) and value(your bucket name)
Buildspec name- Input your buildspec file name ,we user “buildspec.yml” this time.
Check on CloudWatch logs for monitoring build process.
leave all other place as default and click “Continue to Codepipeline”
Created a Codebuild Project and back to codepipeline console.
Project Name is showing the new project which I just created
Create Next
Add deploy stage
Deploy provider — Select AWS CloudFormation
Region — Select your region
Action mode — Create or replace a change set
Stack name — Input your stack name
Change set name — Input your Change set name
Artifact Name — Select BuildArtifact
File name — function-out.yml(which we specified the output file name in the buildspec.yml)
You will see this error but you can ignore it ,because we will create the new stack and change set in the first time of running pipeline.
Capabilities — CAPABILITY_IAM
Role name — Your Role for Cloudformation that we created in the beginning .
And click Next, then Create the pipeline
Add one more stage for CloudFormation .
Please Edit stage under in current Deploy stage.
Click + Add action group under current deploy stage.
Action Provider — AWS CloudFormation
Input artifacts — BuildArtifacts
Action mode — Execute a change set
Stack Name — The same as you input in the first deploy stage.
Change set name — The same as you input in the first deploy stage.
And Save
Codepipeline is working!!
We can see Lambda’s code was updated from Codepipeline and created from CloudFormation.
Congratulation! We finally complicate the whole process for CICD.
Commit from Local→Bitbucket→Codebuild→Cloudformation→Lambda
FAQ:
1.What does Codebuild do?
CodeBuild compiles your source code
runs unit tests
produces artifacts that are ready to deploy.