Setup CI/CD pipeline on AWS with Lambda and the Serverless Framework: Part 2

In the Part 1 of the series, we discussed about how to setup a serverless Continuous Integration part of a CI/CD pipeline leveraging AWS CodePipeline and AWS CodeBuild.

In this article we are going to add the staging deployment stage, a manual approval gate and production deployment stage to our pipeline.

The Serverless Framework

The Serverless Framework is a great cloud provider-independent tool to package and deploy your serverless application. There are other tools out there such as AWS SAM or ClaudiaJS, each one of these with its own pros and cons.

Nevertheless, the Serverless Framework is super easy to use, actively developed and rich of handy plugins. To use it you just need to:

• Install it

npm install serverless -g

• Editing the serverless.yml file

service: gimme-time

provider:
  name: aws
  runtime: nodejs8.10
  environment:
    STAGE: ${opt:stage}

functions:
  gimmetime:
    handler: handler.gimmetime
    events:
      - http:
          path: /time
          method: get
          request:
            parameters:
              querystrings:
                tz: true

This will tell the framework that the service name is gimme-time, the cloud provider is AWS and the run-time is NodeJs 8.10 and sets an environment variable STAGE using the command line options — stage.

Next we define a new function called gimmetime which executes the gimmetime function in the handler.js file (“handler.gimmetime”).

And this function will run whenever there is an HTTP GET request to an HTTP endpoint with path /time which requires a query string parameter tz.

• To deploy it, it’s enough to run

serverless deploy

Easy wasn’t it?! Behind the scenes, the Serverless Framework will creating a new instance of a lambda function named gimmetime, and an API Gateway api and stage to invoke the function. The serverless deploy command will return an URL that you can use to invoke your function.

The Deployment Package

In the last step of the Part 1, we have just instructed the build step of our pipeline to run some code linting and execute the unit tests. We did so by providing the buildspec.yml file along with our source code.

When the build succeeds, we need to create a deployment package that can be used in the deployment stages of our pipeline. And here the packaging feature of the Serverless Framework comes into play.

First, we need to define a directory were the artifacts will be put in and then we need to tell CodeBuild which are our artifacts as output of the build process. To do so we need to edit the buildspec.yml as follows:

version: 0.2

phases:
  install:
    commands:
    - npm install --silent --no-progress -g npm
    - npm install --silent --no-progress -g serverless
  pre_build:
    commands:
    - npm install --silent --no-progress
  build:
    commands:
    - npm run-script lint
    - npm run test
    - mkdir -p target/stg
    - mkdir target/prod
    - serverless package --package target/stg --stage stg -v -r eu-central-1
    - serverless package --package target/prod --stage prod -v -r eu-central-1

artifacts:
  files:
  - target/**/*
  - serverless.yml
  - deploy.sh

Let’s break it down!

1. The install phase it makes sure that npm and serverless framework are both installed.
2. The pre_build phase is used to install the dependencies of our app.
3. The build phase runs the linting and unit tests just as we did in Part 1 and creates a target directory which contains 2 sub directories to hold staging and production artifacts.

The serverless package command is used to generate the deployment packages in the directories target/stg and target/prod. The — stage option will set the STAGE environment variable defined in our serverless.yml allowing us to have a specific behaviour for each stage.

Last but not least, the artifacts section tells CodeBuild to save everything under the target directory, the serverless.yml and the deployment script deploy.sh.

The Deployment Script

The deployment script is a basic bash script that installs the serverless framework on the deployment container and it runs the serverless framework in deployment mode.

#! /bin/bash

npm install -g serverless
serverless deploy --stage $env --package \   $CODEBUILD_SRC_DIR/target/$env -v -r eu-central-1

The $env variable will be initialized in the CodeBuild project we are going to create in a second.

The package option tells Serverless Framework where to find the artifact to deploy. Guess what, the $CODEBUILD_SRC_DIR/target/$env will point exactly to the target/stg or target/prod directory we defined in the previous section.

IAM Role to Deploy Serverless Apps

Before starting to create our stages, It is wort it to spend few words on roles and permissions to run our deployment scripts.

In the Part 1, when we created a CodeBuild project to run our build stage, we had to choose a role to execute it. Back then, we decided to let CodeBuild to create a new role for us. This time we are going to create a new IAM role and policies to execute our deployment stage.

IAM Policy Template for Serverless Apps

As best practice with IAM roles, you should always to go for the least privileged role. We could start writing our policy document by scratch but it is a tedious job, so better use a template generator as a starting point.

We are going to use Yeoman with the handy Serverless Policy Generator, we install them:

npm i -g yo generator-serverless-policy

And we run the Serverless Policy Generator to generate our policy for the staging environment stg:

$ yo serverless-policy
? Your Serverless service name: gimme-time
? You can specify a specific stage, if you like: stg
? You can specify a specific region, if you like: eu-central-1
? Does your service rely on DynamoDB? No
? Is your service going to be using S3 buckets? No
app name gimme-time
app stage stg
app region eu-central-1
Writing to gimme-time-stg-eu-central-1-policy.json

The Serverless service name must match the service name specified in the serverless.yml file.

The file gimme-time-stg-eu-central-1-policy.json contains a generic policy that can be applied to our service. We can use it as starting point to create our tailored policy, for instance, this policy by default allows any Amazon Kinesis operation on a stream called gimme-time-prod-eu-central-1:

{
  "Effect": "Allow",
  "Action": "kinesis:*",
  "Resource": [
    "arn:aws:kinesis:*:*:stream/gimme-time-prod-eu-central-1"
  ]
},

As we do no use kinesis whatsoever, we can safely remove the whole JSON block!

Now that we have our policy JSON nice and clean, we create a new IAM policy and we assign it to a new role, which i called gimme-time-deploy-staging.

Deploying To Staging

Awesome! Now we are ready to add the deploy Staging stage (I am sorry for the word game..) to our Lambda function.

Let go to CodePipeline, select our pipeline and click Edit. Before adding the a new stage, let’s make sure that our Build Stage is storing our artifacts somewhere. Click on Edit Stage in the Build Stage panel, and then on the small “edit” icon, the one that is supposed to represent a paper sheet and a pencil… I guess.. Whatever, this one:

Make sure that you specified an Output artifacts. I named mine BuildArtifact and Save.

The Deployment Stage

Now we can click on Add Stage and provide a name for the new stage, for example DeployStaging, Save and Add Action. Pick an Action Name and select CodeBuild as Action provider.

Just as we did for the Build Stage, we choose a Managed Image with the following properties:

• Operating system: Ubuntu
• Runtime: Node.js
• Runtime version: aws/codebuild/nodejs:8.11.0
• Image version: Always use the latest image

This time we are going to use a service role which we have just created and with the smallest set of permissions to do the job.

Expand the Additional configuration panel and in the Environment variables section I am going to set the variable env to stg.

This is the value that will assume the $env variable in the deploy.sh script we discussed above. In the Buildspec section, choose Insert build Commands and type

bash deploy.sh

We click Save and we should get back to our CodePipeline action

I select BuildArtifact as Input artifact which is the one we created in the Build Stage at the beginning of this paragraph.. Save it and let’s trigger the build clicking on Release Change. Hopefully everything went well and you see every stage green

Troubleshooting

In case your DeployStaging stage is red, do not despair :) Most of the time it’s a problem with permission. Click on the Details link in the DeployStaging panel. It will take you directly in the CodeBuild build project page.

If in your Build logs, the build looks stuck on “Waiting for DOWNLOAD_SOURCE ”

And in the Phase details you see something like

It means that you need to create an IAM policy to allow the Role to access the S3 bucket arn:aws:s3:::gimme-time-artifacts/gimme-time-pipeline/BuildArtif something like this:

{
  "Effect": "Allow",
  "Action": [
    "s3:PutObject",
    "s3:GetObject",
    "s3:DeleteObject"
  ],
  "Resource": [
    "arn:aws:s3:::gimme-time-artifacts/gimme-time-pipeline/BuildArtif/*"
  ]
},

Assign it to the role you specified in the DeployStaging CodeBuild project and run again your pipeline.

Production Approval Gate

We are almost there…

Send Approval Request by Email

Wouldn’t be cool if our pipeline would us send an email with a link to approve the deployment of our project to production? Obviously this step is optional.

To send an email to a one or more recipient we need to setup an Amazon SNS topic. So let’s search for SNS in the AWS console, and click on Create Topic and name it “gimme-time-production-approval”.

Once the topic is created, we need to create an email subscription. Click on Create subscription, choose Email as Protocol and add your email address as Endpoint. At this point, SNS will send you an email with a link in order to confirm the subscription. To notify multiple email addresses, just create more subscriptions to the same topic.

That’s it! We are good to go to add the approval gate.

Manual Approval Gate

To add an Approval Gate we need to edit once again our pipeline. We click on Edit and +Add stage, I named it “ProductionApproval”. Then click on +Add action group, name the action and select Manual Approval as Action provider.

If you want the stage to send the approval requests by email, you need to select in the SNS topic configured in the step above.

Deploying To Production

There we go! We made it to the last step, the production deployment stage. It might sound scary but actually It is simpler than you think… Just take the steps you followed to deploy to staging (see the Deploying To Staging section above) and do it again, replacing stg with prod :) Just to recap:

• Create a new IAM role for CodeBuild just as described above in the IAM Role to Deploy Serverless Apps section, using prod instead of stg
• Add a new pipeline stage named DeployProduction and a new Action creating a new CodeBuild Project just like we did in the The Deployment Stage. We create a new CodeBuild project with the role we’ve just created, and we set prod as env variable in the Environment Variable section

Once you save the new stage and trigger the pipeline, it should look like this.

Congratulations you have a fully functional serverless CI/CD pipeline for your serverless project :)

Are you looking to hire the best software developers, cloud experts, IT consultants.. you name it.. for your project? Touch base with us at www.quantica.io we are looking forward to hearing from you!!!