AWS CloudFormation stack — AWS CloudFormation Stack

Merge audio files with Lambda@Edge/AWS CloudFormation and CI/CD

Published in

shirkalab

9 min readNov 21, 2019

Introduction

The document will explain how to create a CloudFormation stack to deploy a Lambda@Edge function.
The cloudformation template will configure CodePipeLine to deploy the code store inside a CodeCommit repository.
The Lambda@Edge function will join several audio files and store the result on the “data” S3 Bucket.

Project repository: https://github.com/kouz75/lambda-edge-audio

2 minutes guide, go to Part 5 How to install

Why do we want to join audio files ?

With Alexa or Google Assistant, we can insert Speech Synthesis Markup Language (SSML) to improve speech response.
The audio tag lets you provide the URL for an MP3 file. Alexa support 5 audio tag in a response.
For some Alexa skills, we need to use more than 5 audio files, The response can be personalize with the first name of the user, game score, sounds effects etc…

First solution is to pregenerate audio files before releasing the Skill.
Second solution is to generate audio files in real time on users request.

First solution is quite simple to integrate but not really maintainable. You won’t be able to change audio response very easly.

Second solution will allow you to change the audio response without any worry. Another reason to use a single file instead of multiple audio files is the suppression of the latency generated by the use of multiple audio SSML tags.

Part 1 : Resources create by CloudFormation :

Repository — A Git repository in AWS CodeCommit. When you push a change, the pipeline copies the source code into an Amazon S3 bucket and passes it to the build project.
Build project — An AWS CodeBuild build that gets the source code from the pipeline and packages the application. The source includes a build specification with commands that install dependencies and prepare an AWS Serverless Application Model (AWS SAM) template for deployment.
Deployment configuration — The pipeline’s deployment stage defines a set of actions that take the AWS SAM template from the build output, create a change set in AWS CloudFormation, and execute the change set to update the application’s AWS CloudFormation stack.
AWS CloudFormation stack — The deployment stage uses a template to create a stack in AWS CloudFormation. The template is a YAML-formatted document that defines the resources of the Lambda application. The application includes a S3 Bucket, a Lambda function and an CloudFront that invokes it.
Roles — The pipeline, build, and deployment each have a service role that allows them to manage AWS resources. The console creates the pipeline and build roles when you create those resources. You create the role that allows AWS CloudFormation to manage the application stack.

The full CloudFormation template is available on github

Repository and Events Rule

CloudFormation will create a repository which will contains Lambda Code, build spec and SAM template.

A event rule is use to launch the PipeLine each time an update is done to the master branch.

CodeCommitRepo:
  Description: Creating AWS CodeCommit repository for application source code
  Properties:
    RepositoryDescription: !Join
      - ''
      - - !Ref 'ProjectId'
        - ' project repository'
    RepositoryName: !Ref 'RepositoryName'
  Type: AWS::CodeCommit::RepositorySourceEvent:
  Properties:
    Description: Rule for Amazon CloudWatch Events to detect changes to the source repository and trigger pipeline execution
    EventPattern:
      detail:
        event:
          - referenceCreated
          - referenceUpdated
        referenceName:
          - master
        referenceType:
          - branch
      detail-type:
        - CodeCommit Repository State Change
      resources:
        - !GetAtt 'CodeCommitRepo.Arn'
      source:
        - aws.codecommit
    Name: !Join
      - '-'
      - - !Ref 'ProjectId'
        - SourceEvent
    State: ENABLED
    Targets:
      - Arn: !Join [':', ['arn:aws:codepipeline',!Ref 'AWS::Region', !Ref 'AWS::AccountId', !Join ['-', [!Ref 'ProjectId','Pipeline']] ]]
        Id: ProjectPipelineTarget
        RoleArn: !GetAtt 'SourceEventRole.Arn'
  Type: AWS::Events::Rule

2. CodeBuild and Artifact Bucket

CodeBuild is a managed service that compiles your source code, runs tests, produces deployable application artifacts. The artifact will be store in the Artifact S3 Bucket

CodeBuildProject:
  DependsOn:
    - CodeBuildPolicy
  Properties:
    Artifacts:
      Packaging: zip
      Type: codepipeline
    Description: CodeBuild project
    Environment:
      ComputeType: small
      EnvironmentVariables:
        - Name: S3_BUCKET
          Value: !Ref 'ArtifactS3Bucket'
        - Name: AUDIO_FILE_BUCKET
          Value: !Join ['-', [!Ref 'AWS::Region',!Ref 'AWS::AccountId',!Ref 'ProjectId','data']]
      Image: aws/codebuild/nodejs:10.14.1
      Type: container
    Name: !Ref 'ProjectId'
    ServiceRole: !Ref 'CodeBuildRole'
    Source:
      Type: codepipeline
  Type: AWS::CodeBuild::ProjectArtifactS3Bucket:
  Type: AWS::S3::Bucket
  Description: Creating Amazon S3 bucket for AWS CodePipeline artifacts
  Properties:
    BucketName: !Join
      - '-'
      - - !Ref 'AWS::Region'
        - !Ref 'AWS::AccountId'
        - !Ref 'ProjectId'
        - pipe
    Tags:
      - Key: APP
        Value: !Ref 'ProjectId'
    VersioningConfiguration:
      Status: Enabled
    PublicAccessBlockConfiguration:
      BlockPublicAcls: true
      BlockPublicPolicy: true
      IgnorePublicAcls: true
      RestrictPublicBuckets: true

CodeBuild use a buildspec.yml file. A build spec is a collection of build commands and related settings, in YAML format, that CodeBuild uses to run a build.

3. CodePipeline

AWS CodePipeline create a continuous delivery pipeline for the Lambda function. CodePipeline combines source control, build, and deployment resources to create a pipeline that runs whenever you make a change to your application’s source code.
The AWS SAM template file is a YAML or JSON configuration file that adheres to the open source AWS Serverless Application Model specification. You use the template to declare all of the AWS resources that comprise your serverless application.

AWS CodePipeline, deploy to AWS Lambda@edge — Deployment PipeLine

ProjectPipeline:
  DependsOn:
    - LambdaTrustRole
    - CodePipelineTrustRole
    - ArtifactS3Bucket
  Properties:
    ArtifactStore:
      Location: !Ref 'ArtifactS3Bucket'
      Type: S3
    Name: !Join
      - '-'
      - - !Ref 'ProjectId'
        - Pipeline
    RoleArn: !GetAtt
      - CodePipelineTrustRole
      - Arn
    Stages:
      - Actions:
          - ActionTypeId:
              Category: Source
              Owner: AWS
              Provider: CodeCommit
              Version: 1
            Configuration:
              BranchName: master
              PollForSourceChanges: false
              RepositoryName: !Ref 'RepositoryName'
            InputArtifacts: [
            ]
            Name: ApplicationSource
            OutputArtifacts:
              - Name: !Join
                  - '-'
                  - - !Ref 'ProjectId'
                    - SourceArtifact
            RunOrder: 1
        Name: Source
      - Actions:
          - ActionTypeId:
              Category: Build
              Owner: AWS
              Provider: CodeBuild
              Version: 1
            Configuration:
              ProjectName: !Ref 'ProjectId'
            InputArtifacts:
              - Name: !Join
                  - '-'
                  - - !Ref 'ProjectId'
                    - SourceArtifact
            Name: PackageExport
            OutputArtifacts:
              - Name: !Join
                  - '-'
                  - - !Ref 'ProjectId'
                    - BuildArtifact
            RunOrder: 1
        Name: Build
      - Actions:
          - ActionTypeId:
              Category: Deploy
              Owner: AWS
              Provider: CloudFormation
              Version: 1
            Configuration:
              ActionMode: CHANGE_SET_REPLACE
              Capabilities: CAPABILITY_IAM
              ChangeSetName: pipeline-changeset
              ParameterOverrides: !Join
                - ''
                - - '{"ProjectId":"'
                  - !Ref 'ProjectId'
                  - '"}'
              RoleArn: !GetAtt
                - CloudFormationTrustRole
                - Arn
              StackName: !Join
                - '-'
                - - !Ref 'ProjectId'
                  - lambda
              TemplatePath: !Join
                - ''
                - - !Ref 'ProjectId'
                  - -BuildArtifact
                  - ::template-export.yml
            InputArtifacts:
              - Name: !Join
                  - '-'
                  - - !Ref 'ProjectId'
                    - BuildArtifact
            Name: GenerateChangeSet
            OutputArtifacts: [
            ]
            RunOrder: 1
          - ActionTypeId:
              Category: Deploy
              Owner: AWS
              Provider: CloudFormation
              Version: 1
            Configuration:
              ActionMode: CHANGE_SET_EXECUTE
              ChangeSetName: pipeline-changeset
              StackName: !Join
                - '-'
                - - !Ref 'ProjectId'
                  - lambda
            InputArtifacts: [
            ]
            Name: ExecuteChangeSet
            OutputArtifacts: [
            ]
            RunOrder: 2
        Name: Deploy
  Type: AWS::CodePipeline::Pipeline

Part 2: Roles use by the Stack:

LambdaTrustRole:

The role will be use by the Lambda function, it allow the function to :

manage files on the “data” Bucket.
write CloudWatch log
assume lambda and edgelambda role

LambdaTrustRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
                - edgelambda.amazonaws.com
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
        - arn:aws:iam::aws:policy/service-role/AWSConfigRulesExecutionRole
      Path: /
      Policies:
        - PolicyDocument:
            Statement:
              - Action:
                  - logs:CreateLogGroup
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                Effect: Allow
                Resource: '*'
              - Action:
                  - s3:GetObject
                  - s3:PutObject
                  - s3:PutObjectTagging
                Effect: Allow
                Resource: !Join ['', ['arn:aws:s3:::', !Join ['-', [!Ref 'AWS::Region',!Ref 'AWS::AccountId',!Ref 'ProjectId','data']] , /*]]
            Version: 2012-10-17
          PolicyName: LambdaWorkerPolicy
      RoleName: !Join ['-', ['Role',!Ref 'ProjectId', 'Lambda']]

2. CloudFormationTrustRole
The role will be use by CodePipeline, it give CodePipeline access to :

Upload Artifact to S3 bucket
Create and manage Lambda function
Create and manage the “data” bucket
Create and manage Manage CloudFront distribution

CloudFormationTrustRole:
  Type: AWS::IAM::Role
  Description: Creating service role in IAM for AWS CloudFormation
  Properties:
    AssumeRolePolicyDocument:
      Statement:
        - Action: sts:AssumeRole
          Effect: Allow
          Principal:
            Service:
              - cloudformation.amazonaws.com
    Path: /
    Policies:
      - PolicyDocument:
          Statement:
            - Action:
                - s3:PutObject
                - s3:GetObject
                - s3:GetObjectVersion
              Effect: Allow
              Resource:
                - !GetAtt ArtifactS3Bucket.Arn
                - !Join ['', [!GetAtt ArtifactS3Bucket.Arn , '/*']]
            - Action:
                - lambda:*
              Effect: Allow
              Resource: '*'
            - Action:
                - s3:*
              Effect: Allow
              Resource:
                - !Join ['', ['arn:aws:s3:::', !Join ['-', [!Ref 'AWS::Region',!Ref 'AWS::AccountId',!Ref 'ProjectId','data']] ]]
            - Action:
                - cloudfront:*
              Effect: Allow
              Resource: '*'
            - Action:
                - iam:PassRole
              Effect: Allow
              Resource:
                - !GetAtt
                  - LambdaTrustRole
                  - Arn
            - Action:
                - cloudformation:CreateChangeSet
              Effect: Allow
              Resource:
                - arn:aws:cloudformation:us-east-1:aws:transform/Serverless-2016-10-31
        PolicyName: CloudFormationRolePolicy
    RoleName: !Join ['-', ['Role',!Ref 'ProjectId','CloudFormation']]

3. CodeBuildPolicy
The policy will be use by CodeBuild.

Create/Write CloudWatch log
Put/Get object on Artifact and Data Bucket
Pull CodeCommit repository

CodeBuildPolicy:
  Type: AWS::IAM::Policy
  Description: Setting IAM policy for service role for Amazon EC2 instances
  Properties:
    PolicyDocument:
      Statement:
        - Action:
            - logs:CreateLogGroup
            - logs:CreateLogStream
            - logs:PutLogEvents
          Effect: Allow
          Resource: '*'
        - Action:
            - s3:PutObject
            - s3:GetObject
            - s3:GetObjectVersion
          Effect: Allow
          Resource:
            - !GetAtt ArtifactS3Bucket.Arn
            - !Join ['', [!GetAtt ArtifactS3Bucket.Arn , '/*']]
            - !Join ['', ['arn:aws:s3:::', !Join ['-', [!Ref 'AWS::Region',!Ref 'AWS::AccountId',!Ref 'ProjectId','data']] ]]
            - !Join ['', ['arn:aws:s3:::', !Join ['-', [!Ref 'AWS::Region',!Ref 'AWS::AccountId',!Ref 'ProjectId','data']] , '/*']]
        - Action:
            - codecommit:GitPull
          Effect: Allow
          Resource:
            - !Join [':', ['arn:aws:codecommit',!Ref 'AWS::Region', !Ref 'AWS::AccountId',!Ref 'RepositoryName']]
        - Action:
            - kms:GenerateDataKey*
            - kms:Encrypt
            - kms:Decrypt
          Effect: Allow
          Resource:
            - !Join [':', ['arn:aws:kms',!Ref 'AWS::Region', !Ref 'AWS::AccountId','alias/aws/s3']]
    PolicyName: CodeBuildPolicy
    Roles:
      - !Ref 'CodeBuildRole'

4. S3BucketPolicy

The policy allow CloudFront to GetObject from data Bucket

S3BucketPolicy:
  Type: AWS::S3::BucketPolicy
  Properties:
    Bucket: !Ref 'AudioFileS3Bucket'
    PolicyDocument:
      Statement:
        - Effect: Allow
          Principal:
            CanonicalUser:
              Fn::GetAtt: [ CFS3OriginAccessIdentity , S3CanonicalUserId ]
          Action: "s3:GetObject"
          Resource: !Sub "${AudioFileS3Bucket.Arn}/*"

Part 3: Create the Cloudformation Stack

Open CloudFormation on region us-east-1 and click on “Create stack”.

AWS CloudFormation Create stack — Create a CloudFormation stack from a template

AWS CloudFormation Stack spécification — Configure CloudFormation. ProjectId will be use for Lambda/S3/Role Arn

Before creating the stack, don’t forget to confirm that CF will create IAM resources.

After validation, you should wait several minutes for stack to be create.
Once the creation done, you will have to sync the repository create by CloudFormation with github.

Part 4: Lambda deployement configuration

template.yml

AWS SAM templates are an extension of AWS CloudFormation templates.
SAM Template is use during build process. It will be use by CodeBuild to generate a change set for CloudFormation. the pipeline will create:

the lambda function which will be call by Cloudfront

Resources:
  AudioLambdaEdgeFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: !Ref 'ProjectId'
      Handler: index.handler
      Runtime: nodejs10.x
      MemorySize: 3008
      Timeout: 10
      AutoPublishAlias: live
      Role:
        Fn::ImportValue:
          !Join ['-', [!Ref 'ProjectId', !Ref 'AWS::Region', 'LambdaTrustRole']]
      Tags:
        SITE: !Ref 'ProjectId'

the data S3 Bucket which will store audio files
to prevent acces to S3 bucket, public access is disabled, only cloudfront while be allowed to read bucket files.

AudioFileS3Bucket:
    Type: AWS::S3::Bucket
    Description: Creating Amazon S3 bucket to store audio files
    Properties:
      BucketName: !Join ['-', [ !Ref 'AWS::Region', !Ref 'AWS::AccountId', !Ref 'ProjectId', 'data']]
      Tags:
        - Key: APP
          Value: !Ref 'ProjectId'
      VersioningConfiguration:
        Status: Enabled
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: trueS3BucketPolicy:
  Type: AWS::S3::BucketPolicy
  Properties:
    Bucket: !Ref 'AudioFileS3Bucket'
    PolicyDocument:
      Statement:
        - Effect: Allow
          Principal:
            CanonicalUser:
              Fn::GetAtt: [ CFS3OriginAccessIdentity , S3CanonicalUserId ]
          Action: "s3:GetObject"
          Resource: !Sub "${AudioFileS3Bucket.Arn}/*"

CloudFront distribution with an origin access Identity

CFS3Distribution:
  DependsOn: AudioLambdaEdgeFunction
  Type: AWS::CloudFront::Distribution
  Properties:
    DistributionConfig:
      Origins:
        - DomainName: !Join ['', [!Ref 'AudioFileS3Bucket', '.s3.amazonaws.com']]
          Id: myS3Origin
          S3OriginConfig:
            OriginAccessIdentity: !Join ['',['origin-access-identity/cloudfront/', !Ref 'CFS3OriginAccessIdentity'] ]
      Enabled: 'true'
      Comment:  !Ref 'ProjectId'
      DefaultRootObject: index.html
      DefaultCacheBehavior:
        LambdaFunctionAssociations:
          - EventType: origin-response
            LambdaFunctionARN: !Ref AudioLambdaEdgeFunction.Version
        AllowedMethods:
          - GET
          - HEAD
          - OPTIONS
        TargetOriginId: myS3Origin
        ForwardedValues:
          QueryString: 'false'
          Cookies:
            Forward: none
          Headers:
            - Origin
            - Access-Control-Request-Headers
            - Access-Control-Request-Method
        ViewerProtocolPolicy: redirect-to-https
      HttpVersion: http2
      ViewerCertificate:
        CloudFrontDefaultCertificate: true
    Tags:
      - Key: APP
        Value: !Ref 'ProjectId'CFS3OriginAccessIdentity:
  Type: "AWS::CloudFront::CloudFrontOriginAccessIdentity"
  Properties:
    CloudFrontOriginAccessIdentityConfig:
      Comment: !Ref 'ProjectId'

2. buildspec.yml

The buildspec is use by AWS CodeBuild. Several action are define:

install node modules
generate the config file with the bucket name.
Lamdba@edge, Don’t have access to environment variables.
run unit test.
remove unused files to reduce lambda size.
copy assets files to S3
Uploading Artifact to an S3 Bucket and CloudFormation stack with the correct path to the S3

version: 0.2
phases:
  install:
    commands:
      #- npm uninstal aws-sdk
      #- npm uninstal aws-sdk-mock
      - npm install --save-dev


  pre_build:
    commands:
      #generate config file
      - echo {\"audioBucket\"':' \"$AUDIO_FILE_BUCKET\"} > ./config.json
      - npm test
      #removes “extraneous” packages
      - npm prune --production
      #Delete unused binary to reduce package size.
      - rm -rf node_modules/ffmpeg-static/bin/darwin
      - rm -rf node_modules/ffmpeg-static/bin/win32
      - rm -rf node_modules/ffprobe-static/bin/darwin
      - rm -rf node_modules/ffprobe-static/bin/win32
      - rm -rf node_modules/ffmpeg-static/bin/linux/ia32
      - rm -rf node_modules/ffmpeg-static/bin/linux/arm
      - rm -rf node_modules/ffmpeg-static/bin/linux/arm64
      - rm -rf node_modules/ffprobe-static/bin/linux/ia32
      - rm -rf node_modules/ffprobe-static/bin/linux/arm
  build:
    commands:
      - aws s3 cp --recursive ./assets s3://$AUDIO_FILE_BUCKET/assets/
      - aws cloudformation package --template template.yml --s3-bucket $S3_BUCKET --output-template template-export.yml

artifacts:
  type: zip
  files:
    - template-export.yml

Part 5: Install guide

Create a CloudFormation stack with the template cloudformation.yml the stack must be create in us-east-1 region.
Sync this repository with your CodeCommit repository create by CloudFormation
Push your change to your repository. The PipeLine should generate a new version of your lambda function and update CloudFront with the correct Lambda Version.
Try to open the link https://CLOUDFRONT-domain/1-2-1 CLOUDFRONT-domain is the domain name of the CloudFront distribution create by CloudFormation.

If everything works fine, Lambda function should have join files: /assets/1.mp3 with /assets/2.mp3 and /assets/1.mp3 and store the result on S3.
Check your S3 bucket, you will find the file 1–2–1.mp3