AWS CloudFormation stack
AWS CloudFormation stack
AWS CloudFormation Stack

Merge audio files with Lambda@Edge/AWS CloudFormation and CI/CD

Stephane Couzinier
Nov 21 · 9 min read

Introduction

The document will explain how to create a CloudFormation stack to deploy a Lambda@Edge function.
The cloudformation template will configure CodePipeLine to deploy the code store inside a CodeCommit repository.
The Lambda@Edge function will join several audio files and store the result on the “data” S3 Bucket.

Project repository: https://github.com/kouz75/lambda-edge-audio

2 minutes guide, go to Part 5 How to install

Why do we want to join audio files ?

With Alexa or Google Assistant, we can insert Speech Synthesis Markup Language (SSML) to improve speech response.
The audio tag lets you provide the URL for an MP3 file. Alexa support 5 audio tag in a response.
For some Alexa skills, we need to use more than 5 audio files, The response can be personalize with the first name of the user, game score, sounds effects etc…

  • First solution is to pregenerate audio files before releasing the Skill.
  • Second solution is to generate audio files in real time on users request.

First solution is quite simple to integrate but not really maintainable. You won’t be able to change audio response very easly.

Second solution will allow you to change the audio response without any worry. Another reason to use a single file instead of multiple audio files is the suppression of the latency generated by the use of multiple audio SSML tags.

Part 1 : Resources create by CloudFormation :

  • Repository — A Git repository in AWS CodeCommit. When you push a change, the pipeline copies the source code into an Amazon S3 bucket and passes it to the build project.
  • Build project — An AWS CodeBuild build that gets the source code from the pipeline and packages the application. The source includes a build specification with commands that install dependencies and prepare an AWS Serverless Application Model (AWS SAM) template for deployment.
  • Deployment configuration — The pipeline’s deployment stage defines a set of actions that take the AWS SAM template from the build output, create a change set in AWS CloudFormation, and execute the change set to update the application’s AWS CloudFormation stack.
  • AWS CloudFormation stack — The deployment stage uses a template to create a stack in AWS CloudFormation. The template is a YAML-formatted document that defines the resources of the Lambda application. The application includes a S3 Bucket, a Lambda function and an CloudFront that invokes it.
  • Roles — The pipeline, build, and deployment each have a service role that allows them to manage AWS resources. The console creates the pipeline and build roles when you create those resources. You create the role that allows AWS CloudFormation to manage the application stack.

The full CloudFormation template is available on github

  1. Repository and Events Rule

CloudFormation will create a repository which will contains Lambda Code, build spec and SAM template.

A event rule is use to launch the PipeLine each time an update is done to the master branch.

CodeCommitRepo:
Description: Creating AWS CodeCommit repository for application source code
Properties:
RepositoryDescription: !Join
- ''
- - !Ref 'ProjectId'
- ' project repository'
RepositoryName: !Ref 'RepositoryName'
Type: AWS::CodeCommit::Repository
SourceEvent:
Properties:
Description: Rule for Amazon CloudWatch Events to detect changes to the source repository and trigger pipeline execution
EventPattern:
detail:
event:
- referenceCreated
- referenceUpdated
referenceName:
- master
referenceType:
- branch
detail-type:
- CodeCommit Repository State Change
resources:
- !GetAtt 'CodeCommitRepo.Arn'
source:
- aws.codecommit
Name: !Join
- '-'
- - !Ref 'ProjectId'
- SourceEvent
State: ENABLED
Targets:
- Arn: !Join [':', ['arn:aws:codepipeline',!Ref 'AWS::Region', !Ref 'AWS::AccountId', !Join ['-', [!Ref 'ProjectId','Pipeline']] ]]
Id: ProjectPipelineTarget
RoleArn: !GetAtt 'SourceEventRole.Arn'
Type: AWS::Events::Rule

2. CodeBuild and Artifact Bucket

CodeBuild is a managed service that compiles your source code, runs tests, produces deployable application artifacts. The artifact will be store in the Artifact S3 Bucket

CodeBuildProject:
DependsOn:
- CodeBuildPolicy
Properties:
Artifacts:
Packaging: zip
Type: codepipeline
Description: CodeBuild project
Environment:
ComputeType: small
EnvironmentVariables:
- Name: S3_BUCKET
Value: !Ref 'ArtifactS3Bucket'
- Name: AUDIO_FILE_BUCKET
Value: !Join ['-', [!Ref 'AWS::Region',!Ref 'AWS::AccountId',!Ref 'ProjectId','data']]
Image: aws/codebuild/nodejs:10.14.1
Type: container
Name: !Ref 'ProjectId'
ServiceRole: !Ref 'CodeBuildRole'
Source:
Type: codepipeline
Type: AWS::CodeBuild::Project
ArtifactS3Bucket:
Type: AWS::S3::Bucket
Description: Creating Amazon S3 bucket for AWS CodePipeline artifacts
Properties:
BucketName: !Join
- '-'
- - !Ref 'AWS::Region'
- !Ref 'AWS::AccountId'
- !Ref 'ProjectId'
- pipe
Tags:
- Key: APP
Value: !Ref 'ProjectId'
VersioningConfiguration:
Status: Enabled
PublicAccessBlockConfiguration:
BlockPublicAcls: true
BlockPublicPolicy: true
IgnorePublicAcls: true
RestrictPublicBuckets: true

CodeBuild use a buildspec.yml file. A build spec is a collection of build commands and related settings, in YAML format, that CodeBuild uses to run a build.

3. CodePipeline

AWS CodePipeline create a continuous delivery pipeline for the Lambda function. CodePipeline combines source control, build, and deployment resources to create a pipeline that runs whenever you make a change to your application’s source code.
The AWS SAM template file is a YAML or JSON configuration file that adheres to the open source AWS Serverless Application Model specification. You use the template to declare all of the AWS resources that comprise your serverless application.

AWS CodePipeline, deploy to AWS Lambda@edge
AWS CodePipeline, deploy to AWS Lambda@edge
Deployement PipeLine
ProjectPipeline:
DependsOn:
- LambdaTrustRole
- CodePipelineTrustRole
- ArtifactS3Bucket
Properties:
ArtifactStore:
Location: !Ref 'ArtifactS3Bucket'
Type: S3
Name: !Join
- '-'
- - !Ref 'ProjectId'
- Pipeline
RoleArn: !GetAtt
- CodePipelineTrustRole
- Arn
Stages:
- Actions:
- ActionTypeId:
Category: Source
Owner: AWS
Provider: CodeCommit
Version: 1
Configuration:
BranchName: master
PollForSourceChanges: false
RepositoryName: !Ref 'RepositoryName'
InputArtifacts: [
]
Name: ApplicationSource
OutputArtifacts:
- Name: !Join
- '-'
- - !Ref 'ProjectId'
- SourceArtifact
RunOrder: 1
Name: Source
- Actions:
- ActionTypeId:
Category: Build
Owner: AWS
Provider: CodeBuild
Version: 1
Configuration:
ProjectName: !Ref 'ProjectId'
InputArtifacts:
- Name: !Join
- '-'
- - !Ref 'ProjectId'
- SourceArtifact
Name: PackageExport
OutputArtifacts:
- Name: !Join
- '-'
- - !Ref 'ProjectId'
- BuildArtifact
RunOrder: 1
Name: Build
- Actions:
- ActionTypeId:
Category: Deploy
Owner: AWS
Provider: CloudFormation
Version: 1
Configuration:
ActionMode: CHANGE_SET_REPLACE
Capabilities: CAPABILITY_IAM
ChangeSetName: pipeline-changeset
ParameterOverrides: !Join
- ''
- - '{"ProjectId":"'
- !Ref 'ProjectId'
- '"}'
RoleArn: !GetAtt
- CloudFormationTrustRole
- Arn
StackName: !Join
- '-'
- - !Ref 'ProjectId'
- lambda
TemplatePath: !Join
- ''
- - !Ref 'ProjectId'
- -BuildArtifact
- ::template-export.yml
InputArtifacts:
- Name: !Join
- '-'
- - !Ref 'ProjectId'
- BuildArtifact
Name: GenerateChangeSet
OutputArtifacts: [
]
RunOrder: 1
- ActionTypeId:
Category: Deploy
Owner: AWS
Provider: CloudFormation
Version: 1
Configuration:
ActionMode: CHANGE_SET_EXECUTE
ChangeSetName: pipeline-changeset
StackName: !Join
- '-'
- - !Ref 'ProjectId'
- lambda
InputArtifacts: [
]
Name: ExecuteChangeSet
OutputArtifacts: [
]
RunOrder: 2
Name: Deploy
Type: AWS::CodePipeline::Pipeline

Part 2: Roles use by the Stack:

  1. LambdaTrustRole:

The role will be use by the Lambda function, it allow the function to :

  • manage files on the “data” Bucket.
  • write CloudWatch log
  • assume lambda and edgelambda role
LambdaTrustRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: sts:AssumeRole
Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
- edgelambda.amazonaws.com
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
- arn:aws:iam::aws:policy/service-role/AWSConfigRulesExecutionRole
Path: /
Policies:
- PolicyDocument:
Statement:
- Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
Effect: Allow
Resource: '*'
- Action:
- s3:GetObject
- s3:PutObject
- s3:PutObjectTagging
Effect: Allow
Resource: !Join ['', ['arn:aws:s3:::', !Join ['-', [!Ref 'AWS::Region',!Ref 'AWS::AccountId',!Ref 'ProjectId','data']] , /*]]
Version: 2012-10-17
PolicyName: LambdaWorkerPolicy
RoleName: !Join ['-', ['Role',!Ref 'ProjectId', 'Lambda']]

2. CloudFormationTrustRole
The role will be use by CodePipeline, it give CodePipeline access to :

  • Upload Artifact to S3 bucket
  • Create and manage Lambda function
  • Create and manage the “data” bucket
  • Create and manage Manage CloudFront distribution
CloudFormationTrustRole:
Type: AWS::IAM::Role
Description: Creating service role in IAM for AWS CloudFormation
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: sts:AssumeRole
Effect: Allow
Principal:
Service:
- cloudformation.amazonaws.com
Path: /
Policies:
- PolicyDocument:
Statement:
- Action:
- s3:PutObject
- s3:GetObject
- s3:GetObjectVersion
Effect: Allow
Resource:
- !GetAtt ArtifactS3Bucket.Arn
- !Join ['', [!GetAtt ArtifactS3Bucket.Arn , '/*']]
- Action:
- lambda:*
Effect: Allow
Resource: '*'
- Action:
- s3:*
Effect: Allow
Resource:
- !Join ['', ['arn:aws:s3:::', !Join ['-', [!Ref 'AWS::Region',!Ref 'AWS::AccountId',!Ref 'ProjectId','data']] ]]
- Action:
- cloudfront:*
Effect: Allow
Resource: '*'
- Action:
- iam:PassRole
Effect: Allow
Resource:
- !GetAtt
- LambdaTrustRole
- Arn
- Action:
- cloudformation:CreateChangeSet
Effect: Allow
Resource:
- arn:aws:cloudformation:us-east-1:aws:transform/Serverless-2016-10-31
PolicyName: CloudFormationRolePolicy
RoleName: !Join ['-', ['Role',!Ref 'ProjectId','CloudFormation']]

3. CodeBuildPolicy
The policy will be use by CodeBuild.

  • Create/Write CloudWatch log
  • Put/Get object on Artifact and Data Bucket
  • Pull CodeCommit repository
CodeBuildPolicy:
Type: AWS::IAM::Policy
Description: Setting IAM policy for service role for Amazon EC2 instances
Properties:
PolicyDocument:
Statement:
- Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
Effect: Allow
Resource: '*'
- Action:
- s3:PutObject
- s3:GetObject
- s3:GetObjectVersion
Effect: Allow
Resource:
- !GetAtt ArtifactS3Bucket.Arn
- !Join ['', [!GetAtt ArtifactS3Bucket.Arn , '/*']]
- !Join ['', ['arn:aws:s3:::', !Join ['-', [!Ref 'AWS::Region',!Ref 'AWS::AccountId',!Ref 'ProjectId','data']] ]]
- !Join ['', ['arn:aws:s3:::', !Join ['-', [!Ref 'AWS::Region',!Ref 'AWS::AccountId',!Ref 'ProjectId','data']] , '/*']]
- Action:
- codecommit:GitPull
Effect: Allow
Resource:
- !Join [':', ['arn:aws:codecommit',!Ref 'AWS::Region', !Ref 'AWS::AccountId',!Ref 'RepositoryName']]
- Action:
- kms:GenerateDataKey*
- kms:Encrypt
- kms:Decrypt
Effect: Allow
Resource:
- !Join [':', ['arn:aws:kms',!Ref 'AWS::Region', !Ref 'AWS::AccountId','alias/aws/s3']]
PolicyName: CodeBuildPolicy
Roles:
- !Ref 'CodeBuildRole'

4. S3BucketPolicy

The policy allow CloudFront to GetObject from data Bucket

S3BucketPolicy:
Type: AWS::S3::BucketPolicy
Properties:
Bucket: !Ref 'AudioFileS3Bucket'
PolicyDocument:
Statement:
- Effect: Allow
Principal:
CanonicalUser:
Fn::GetAtt: [ CFS3OriginAccessIdentity , S3CanonicalUserId ]
Action: "s3:GetObject"
Resource: !Sub "${AudioFileS3Bucket.Arn}/*"

Part 3: Create the Cloudformation Stack

Open CloudFormation on region us-east-1 and click on “Create stack”.

AWS CloudFormation Create stack
AWS CloudFormation Create stack
Create a CloudFormation stack from a template
AWS CloudFormation Stack spécification
AWS CloudFormation Stack spécification
Configure CloudFormation. ProjectId will be use for Lambda/S3/Role Arn
Before creating the stack, don’t forget to confirm that CF will create IAM resources.

After validation, you should wait several minutes for stack to be create.
Once the creation done, you will have to sync the repository create by CloudFormation with github.

Part 4: Lambda deployement configuration

  1. template.yml

AWS SAM templates are an extension of AWS CloudFormation templates.
SAM Template is use during build process. It will be use by CodeBuild to generate a change set for CloudFormation. the pipeline will create:

  • the lambda function which will be call by Cloudfront
Resources:
AudioLambdaEdgeFunction:
Type: AWS::Serverless::Function
Properties:
FunctionName: !Ref 'ProjectId'
Handler: index.handler
Runtime: nodejs10.x
MemorySize: 3008
Timeout: 10
AutoPublishAlias: live
Role:
Fn::ImportValue:
!Join ['-', [!Ref 'ProjectId', !Ref 'AWS::Region', 'LambdaTrustRole']]
Tags:
SITE: !Ref 'ProjectId'
  • the data S3 Bucket which will store audio files
    to prevent acces to S3 bucket, public access is disabled, only cloudfront while be allowed to read bucket files.
AudioFileS3Bucket:
Type: AWS::S3::Bucket
Description: Creating Amazon S3 bucket to store audio files
Properties:
BucketName: !Join ['-', [ !Ref 'AWS::Region', !Ref 'AWS::AccountId', !Ref 'ProjectId', 'data']]
Tags:
- Key: APP
Value: !Ref 'ProjectId'
VersioningConfiguration:
Status: Enabled
PublicAccessBlockConfiguration:
BlockPublicAcls: true
BlockPublicPolicy: true
IgnorePublicAcls: true
RestrictPublicBuckets: true
S3BucketPolicy:
Type: AWS::S3::BucketPolicy
Properties:
Bucket: !Ref 'AudioFileS3Bucket'
PolicyDocument:
Statement:
- Effect: Allow
Principal:
CanonicalUser:
Fn::GetAtt: [ CFS3OriginAccessIdentity , S3CanonicalUserId ]
Action: "s3:GetObject"
Resource: !Sub "${AudioFileS3Bucket.Arn}/*"
  • CloudFront distribution with an origin access Identity
CFS3Distribution:
DependsOn: AudioLambdaEdgeFunction
Type: AWS::CloudFront::Distribution
Properties:
DistributionConfig:
Origins:
- DomainName: !Join ['', [!Ref 'AudioFileS3Bucket', '.s3.amazonaws.com']]
Id: myS3Origin
S3OriginConfig:
OriginAccessIdentity: !Join ['',['origin-access-identity/cloudfront/', !Ref 'CFS3OriginAccessIdentity'] ]
Enabled: 'true'
Comment: !Ref 'ProjectId'
DefaultRootObject: index.html
DefaultCacheBehavior:
LambdaFunctionAssociations:
- EventType: origin-response
LambdaFunctionARN: !Ref AudioLambdaEdgeFunction.Version
AllowedMethods:
- GET
- HEAD
- OPTIONS
TargetOriginId: myS3Origin
ForwardedValues:
QueryString: 'false'
Cookies:
Forward: none
Headers:
- Origin
- Access-Control-Request-Headers
- Access-Control-Request-Method
ViewerProtocolPolicy: redirect-to-https
HttpVersion: http2
ViewerCertificate:
CloudFrontDefaultCertificate: true
Tags:
- Key: APP
Value: !Ref 'ProjectId'
CFS3OriginAccessIdentity:
Type: "AWS::CloudFront::CloudFrontOriginAccessIdentity"
Properties:
CloudFrontOriginAccessIdentityConfig:
Comment: !Ref 'ProjectId'

2. buildspec.yml

The buildspec is use by AWS CodeBuild. Several action are define:

  • install node modules
  • generate the config file with the bucket name.
    Lamdba@edge, Don’t have access to environment variables.
  • run unit test.
  • remove unused files to reduce lambda size.
  • copy assets files to S3
  • Uploading Artifact to an S3 Bucket and CloudFormation stack with the correct path to the S3
version: 0.2
phases:
install:
commands:
#- npm uninstal aws-sdk
#- npm uninstal aws-sdk-mock
- npm install --save-dev


pre_build:
commands:
#generate config file
- echo {\"audioBucket\"':' \"$AUDIO_FILE_BUCKET\"} > ./config.json
- npm test
#removes “extraneous” packages
- npm prune --production
#Delete unused binary to reduce package size.
- rm -rf node_modules/ffmpeg-static/bin/darwin
- rm -rf node_modules/ffmpeg-static/bin/win32
- rm -rf node_modules/ffprobe-static/bin/darwin
- rm -rf node_modules/ffprobe-static/bin/win32
- rm -rf node_modules/ffmpeg-static/bin/linux/ia32
- rm -rf node_modules/ffmpeg-static/bin/linux/arm
- rm -rf node_modules/ffmpeg-static/bin/linux/arm64
- rm -rf node_modules/ffprobe-static/bin/linux/ia32
- rm -rf node_modules/ffprobe-static/bin/linux/arm
build:
commands:
- aws s3 cp --recursive ./assets s3://$AUDIO_FILE_BUCKET/assets/
- aws cloudformation package --template template.yml --s3-bucket $S3_BUCKET --output-template template-export.yml

artifacts:
type: zip
files:
- template-export.yml

Part 5: Install guide

  • Create a CloudFormation stack with the template cloudformation.yml the stack must be create in us-east-1 region.
  • Sync this repository with your CodeCommit repository create by CloudFormation
  • Push your change to your repository. The PipeLine should generate a new version of your lambda function and update CloudFront with the correct Lambda Version.
  • Try to open the link https://CLOUDFRONT-domain/1-2-1 CLOUDFRONT-domain is the domain name of the CloudFront distribution create by CloudFormation.

If everything works fine, Lambda function should have join files: /assets/1.mp3 with /assets/2.mp3 and /assets/1.mp3 and store the result on S3.
Check your S3 bucket, you will find the file 1–2–1.mp3

shirkalab

ShirkaLAB is a voice application agency. We design, create, develop, host and manage custom Alexa skills for leading brands and publishers. ShirkaLAB helps publishers to track and monetize their skills and actions on Google

Stephane Couzinier

Written by

CTO of ShirkaLAB Agency

shirkalab

shirkalab

ShirkaLAB is a voice application agency. We design, create, develop, host and manage custom Alexa skills for leading brands and publishers. ShirkaLAB helps publishers to track and monetize their skills and actions on Google

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade