CodePipeline + CloudFormation + Lambda
Deploying CodePipeline solution for Python and Java Lambda using CloudFormation
Hi,
My name is Raul Barreto. I am a certified AWS Solutions Architect Associate and I’m now studying for taking the exam Solutions Architect Professional next month (Mar/2020).
One of the AWS solutions that I am struggling with is the AWS Developer Tools (CodeBuild, CodePipeline and CodeDeploy). So I decided to actually implement a simple but well covered CodePipeline with CloudFormation and Lambda.
I have previous knowledge in building CloudFormation templates and Lambda functions using mostly Python and Java.
Understanding AWS Developer Tools
CodePipeline = CodeCommit -> CodeBuild -> Deploy
If you never heard about the AWS Developer Tools, I will briefly talk about this set of tools.
AWS CodeCommit
This service/tool provides you with a private git repository. Much like Bitbucket or GitHub. You just create a new repository (like a new project in GitHub), clone the repository in your computer and start to put files in there using git commands like commit, push and pull.
The credentials solutions in AWS for CodeCommit is HTTPS Git credentials inside IAM User. If you have any difficulties to create this credentials, take a look in the CodeCommit user guide at https://docs.aws.amazon.com/codecommit/latest/userguide/setting-up-gc.html. Be aware that you can have up to 5 users with Git Credentials for free in AWS and pay $1 for any additional user.
So you created the repository, put your code on it, developed and tested in your favourite IDE but that’s not how you want to deploy the code in your services. You some times need it to be compiled, like Java, or to be packaged with the needed libraries, like Python. That’s the job of the next service.
AWS CodeBuild
The CodeBuild service allows you to get something (usually your code) from an origin and generates for you an output (usually a file).
To tell the service how it can generate output for you, you need to specify a buildspec.yml, within your project, with the specifications and commands to generate the desired output. It’s much like the generation of a Docker Image with a Dockerfile or the generation of EBExtension commands for Beanstalk.
Below is the generated buildspec.yml that I generate for my Python code:
You can notice that it has a list of runtime versions to run. You can specify your code language version in this section of the buildspec without worrying about how the installation of Python will occur.
Next you have pre_build commands to do some installations or preparations before the actual build. With Python, I tell the buildspec to install the Python libraries from requirements.txt in the same folder from the code. After this, it will remove buildspec.yml and .gitignore since I don't need this to run the Python Code. The last step from pre_build is to move the sam.yaml (AWS Serverless Applicaton Model) file to /tmp/ folder because it will need this file to generate the output.
In the build section, I simply zip all the files and libraries folders to /tmp/output.zip, and next step is to use the sam.yaml model to generate a CloudFormation package that will get the /tmp/output.zip, put it in the S3 Bucket defined in the $S3_BUCKET environment variable and generate a file called template.yaml. This is the template file that CloudFormation will use to create the Lambda Function with the code generated in /tmp/output.zip.
For whom don’t know about the SAM files, it’s a simplification for CloudFormation to create a serverless application within AWS. It can generate API Gateway, SQS queues, Lambda Functions with little effort.
Take a look in the sam.yaml file:
It has a parameter called Name, that is the name of Lambda Function. And I define some parameters for this Lambda Function.
I tell where is the code for this function (remember that the output from buildspec.yml is /tmp/output.zip), what Runtime to run, what is the memory size and timeout in seconds. With this little code, it can create a CloudFormation template that creates the Lambda Function with the minimum needs to run.
But after you compiled or packaged your application and generated a CloudFormation script, you need to run it. In the next step, you need the next service to tie up all this complex input and output files.
AWS CodePipeline
Basically CodePipeline receives some input, it can be your code on CodeCommit, BitBucket or GitHub, or some file in S3; and generates an output or the desired state. It also can have some manual approval between state to manual authorize a production deployment or it can be automatic to deploy in a staging environment.
The structure that I created for this test is shown below:
It has as source the CodeCommit repository that I created with the stage name Source. The code from CodeCommit is an input for the next stage, the Build stage. This Build stage calls the CodeBuild service and generates a CloudFormation template as output and this template is output for the Deploy stage. It could have a test stage to do some automatic testing or a staging deploy to deploy the build in stating environment but I wanted to keep it simple.
The Source and Build stages are the CodeCommit and CodeBuild service as I wrote above but the Deploy stage is very easy because CodePipeline handles to you the CloudFormation deployments.
CodePipeline supports a few types of CloudFormation deployment, in this test I used CHANGE_SET_REPLACE in Deploy action (it only validates the CloudFormation template and don’t create the resources) and CHANGE_SET_EXECUTE in ExecuteChangeSet action(this command creates the CloudFormation resources).
CloudWatch Rule Event
Last but not least, you have to setup a CloudWatch Rule Event to trigger your CodePipeline pipeline when there’s a new release in CodeCommit repository. This will very important piece to create the start the automation process, without it you would have to call the CodePipeline start execution process via console or command lines.
CloudFormation
So, I talked about the AWS Developer Tools and now you have some idea of how to implement and deploy a test solution as I did. But what if you want to deploy dozens or hundreds of lambda or microservices solutions; how to keep/maintain all these DevOps workflows with CodePipeline ?
The solution is to use CloudFormation to create a bunch of templates that cover all of your needs.
You may not notice so far, but all these services from AWS Developer Tools services roles, execution roles, policy permissions, environment variables, input/output from CodePipeline … To keep all these connections secured and replicable you have to use some Infrastructure as Code; it can be CloudFormation, Terraform or Ansible. In this case, I’m using the CloudFormation templates to do the job.
If you look at CloudFormation graph, you can see the relationships between all the services. The template creates 4 IAM roles (1 for CodeBuild, 1 for CloudWatch Event Rule, 1 for CodePipeline and 1 for CodePipeline to execute CloudFormation SAM templates), 1 CodeBuild Project, 1 CodePipeline pipeline, 1 CloudWatch Event Rule and 1 S3 Bucket that’s used for the services.
Since I build this CloudFormation template and everything is working as it should, I started to change the template and try other solutions. I create 2 templates for 2 programming languages, Python and Java, with manual approval or automatic approval. My Java solution is using Maven to package the code into a .jar but you can change it to use other solutions, like standalone .jar libraries.
Feel if you free to contact me in any doubt.