Setting up CI/CD for Lambda Functions using AWS CodePipeline+BitBucket+CloudFormation

How to deploy AWS Lambda functions of Python using Bitbucket, CodeBuild, and CloudFormation and orchestrated by CodePipeline.

This is my Architecture diagram:

Prerequisite:

1. Create a BitBucket Account and a branch
2. Create a bucket for packaged source code
3. Create a bucket for packaged Cloudformation template file

Permission:

• Lambda IAM Role

{
    "Statement": [
        {
            "Action": [
                "apigateway:*",
                "codedeploy:*",
                "lambda:*",
                "cloudformation:CreateChangeSet",
                "iam:GetRole",
                "iam:CreateRole",
                "iam:DeleteRole",
                "iam:PutRolePolicy",
                "iam:AttachRolePolicy",
                "iam:DeleteRolePolicy",
                "iam:DetachRolePolicy",
                "iam:PassRole",
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:GetBucketVersioning"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ],
    "Version": "2012-10-17"
}

• CodePipeline IAM Role: Create by Console
• CodeBuild IAM Role

It will be use on Build stage setting.

Please create and attach this policy to your codebuild role after you created the Codebuild role on console.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ListObjectsInBucket",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::your-default-artifact-store-bucket",
                "arn:aws:s3:::your-sourcecode-store-bucket"
]
        },
        {
            "Sid": "AllObjectActions",
            "Effect": "Allow",
            "Action": "s3:*Object",
            "Resource": [
                "arn:aws:s3:::your-artifact-store-bucket/*"
                "arn:aws:s3:::your-sourcecode-store-bucket/*
            ]
        }
    ]
}

• CloudFormation IAM Role

It will be use on Deploy stage for CloudFormation.

{
    "Statement": [
        {
            "Action": [
                "codedeploy:*",
                "lambda:*",
                "cloudformation:CreateChangeSet",
                "iam:GetRole",
                "iam:CreateRole",
                "iam:DeleteRole",
                "iam:PutRolePolicy",
                "iam:AttachRolePolicy",
                "iam:DeleteRolePolicy",
                "iam:DetachRolePolicy",
                "iam:PassRole",
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:GetBucketVersioning"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ],
    "Version": "2012-10-17"
}

Steps:

1. CLONE repository and PUSH source code, build template, and deploy template files to the repository
2. Create pipeline in Codepipeline
3. Create Codebuild project
4. Source code is deployed by CodePipeline automatically

1. CLONE repository and PUSH source code, build template, and deploy template files to the repository

My file is as below in BitBucket:

• buildspec.yml

1.This is the yml to tell Codebuild how to build it.

2. install: define runtime version

3. build: install some external library

4. post_build: The output file would be in the artifact zip that is passed by CodePipeline from CodeBuild to CloudFormation action.

version: 0.2
phases:
  install:
    runtime-versions:
      python: 3.7
  build:
    commands:
      - echo "install pandas and numpy"
      - mkdir dep
      - pip install pandas -t ./dep
      - pip install numpy -t ./dep
  post_build:
    commands:
      - aws cloudformation package --template-file function.yml --s3-bucket $S3BUCKET --force-upload --output-template-file function-out.yml
artifacts:
  type: zip
  files:
    - function.yml
    - function-out.yml #Use this template for Deploy stage

• template.yml

I use a SAM template for CodeBuild here. You should define the value as below,Codebuild will compile source code:

1. CodeUri: your code path
2. FunctionName: Lambda Function Name
3. Handler: Your lambda handler

Resources:
SampleLambdaFunction:
Properties:
CodeUri: . #Directory the source file is
FunctionName: cicd-lambda-print_log
Handler: lambda_function.lambda_handler
Runtime: python3.7
Type: AWS::Serverless::Function
Transform: AWS::Serverless-2016-10-31

• lambda_function.py

My Lambda Function is as below:

import numpy as np
import pandas as pd
def lambda_handler(event, context):
    return 'commit from 20201222 14:03'

And we can push these three file to the repo by using the cmd on the local repo.

git add .
git commit -m "first commit"
git push

2.Create pipeline in Codepipeline

Click “Create Pipeline”

Choose pipeline settings

Pipeline Name- Testpipeline

Service role- New Service role and check

Advance settings- Custom location: Specified a S3 bucket for packaged Cloudformation template file. The output file would be in the artifact zip that is passed by CodePipeline from CodeBuild to CloudFormation action.

Add source stage

Select “Bitbucket” and “connect to Bitbucket”

Create a connection to Bitbucket

Input Connection Name and connect Bitbucket

Grant access to AWS codestar

1.Click “Install a new app”

2.Connect

Authorize for codestar to read info as below

Connection will be input in some value and Click Connect

Go to Bitbucket>Personal Setting to check the link between codestar and Bitbucket

Add source stage

Select your repo and Branch on Bitbucket

Add Build stage

Build provider -AWS CodeBuild

Region-Your region

Project name- Create project and move to another page as below

Create build project

Project Name- Your project name

Environment image- Managed image

Operating system- Ubuntu

Runtimes- Standard

Image- aws/codebuild/standard:3.0

Environment type- Linux

Service Role- check

Environment variables- Input “S3BUCKET” (The variables that you specified in the buildspec.yml) and value(your bucket name)

Buildspec name- Input your buildspec file name ,we user “buildspec.yml” this time.

Check on CloudWatch logs for monitoring build process.

leave all other place as default and click “Continue to Codepipeline”

Created a Codebuild Project and back to codepipeline console.

Project Name is showing the new project which I just created

Create Next

Add deploy stage

Deploy provider — Select AWS CloudFormation

Region — Select your region

Action mode — Create or replace a change set

Stack name — Input your stack name

Change set name — Input your Change set name

Artifact Name — Select BuildArtifact

File name — function-out.yml(which we specified the output file name in the buildspec.yml)

You will see this error but you can ignore it ,because we will create the new stack and change set in the first time of running pipeline.

Capabilities — CAPABILITY_IAM

Role name — Your Role for Cloudformation that we created in the beginning .

And click Next, then Create the pipeline

Add one more stage for CloudFormation .

Please Edit stage under in current Deploy stage.

Click + Add action group under current deploy stage.

Action Provider — AWS CloudFormation

Input artifacts — BuildArtifacts

Action mode — Execute a change set

Stack Name — The same as you input in the first deploy stage.

Change set name — The same as you input in the first deploy stage.

And Save

Codepipeline is working!!

We can see Lambda’s code was updated from Codepipeline and created from CloudFormation.

Congratulation! We finally complicate the whole process for CICD.

Commit from Local→Bitbucket→Codebuild→Cloudformation→Lambda

FAQ:

1.What does Codebuild do?

CodeBuild compiles your source code

runs unit tests

produces artifacts that are ready to deploy.