Creating AWS CodePipeline through AWS CloudFormation for cross region deployment to a CloudFormation stack.
INTRODUCTION
As an AWS Cloud Enthusiast and someone who works on AWS Cloud on a regular basis, I came across a very common use case where people use AWS CodePipeline to deploy/manage a CloudFormation stack in another region. As an example, consider my CodePipeline that will be created in region-1 and it will deploy/manage the CloudFormation stack in region-2.
I have observed that many users make use of CloudFormation to set up this architecture. Hence, I have written this article to help people understand how this use case can be achieved with a simple example. I have highlighted some important aspects and points where anyone would usually go wrong while setting up this infrastructure using CloudFormation.
Source Code for this article is available here — SOURCE CODE
Before I go into the details of the implementation , I would like to provide some introduction to AWS CodePipeline and AWS CloudFormation which are the prerequisites to understand this article along with some basic hands-on experience with AWS.
WHAT IS AWS CODEPIPELINE ?
Continuous integration, delivery, and deployment, known collectively as CI/CD, is an integral part of modern development intended to reduce errors during integration and deployment while increasing project velocity. CI/CD is a philosophy and set of practices often augmented by robust tooling that emphasize automated testing at each stage of the software pipeline.
AWS offers a bunch of Code Suite services that help set up CI/CD with AWS Cloud. AWS CodePipeline is one such service which is a fully managed continuous delivery service that helps you automate your release pipelines for fast and reliable application and infrastructure updates. AWS CodePipeline is a continuous delivery service you can use to model, visualize, and automate the steps required to release your software. You can quickly model and configure the different stages of a software release process. CodePipeline automates the steps required to release your software changes continuously. Read more about CodePipeline here — https://docs.aws.amazon.com/codepipeline/latest/userguide/welcome.html
WHAT IS CLOUDFORMATION ?
AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less time managing those resources and more time focusing on your applications that run in AWS. You create a template that describes all the AWS resources that you want (like Amazon EC2 instances or Amazon RDS DB instances), and AWS CloudFormation takes care of provisioning and configuring those resources for you.
AWS CloudFormation allows you to use programming languages or a simple text file to model and provision, in an automated and secure manner, all the resources needed for your applications across all regions and accounts. This gives you a single source of truth for your AWS and third party resources. More details on AWS CloudFormation — https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html
USE CASE DESCRIPTION:
This article will help you create a CloudFormation stack to set up a pipeline using AWS CodePipeline in region-1, which will be responsible to deploy a CloudFormation stack in another region ‘region-2’ using the source code (template file and parameter configuration file) stored in a S3 bucket. Setting up the Pipeline through the AWS Console is a very simple, however it is a bit challenging and requires us to keep a note of certain important things when we create this through AWS CloudFormation.
CODEPIPELINE CONFIGURATION:
You will be creating a simple Pipeline with only two stages to keep it simple and generic. You can add additional stages to the Pipeline by following the CodePipeline Structure Reference.
Source Stage :
You have to store the source code which is required for the creation of the CloudFormation stack (template file and parameter configuration file) in an S3 bucket in the region where the CodePipeline needs to be created. You can store your source code in CodeCommit, GitHub or BitBucket. The source code (CodepipelineCrossRegion.zip) must consist of the template file (sampleEC2.yaml) and the parameter file (parameter-configuration.json).
NOTE : The parameter-configuration.json needs to be modified based the resources in your account. Also the main template needs to be modified if you are using a different source provider.
Deploy stage :
Since the goal is to deploy a CloudFormation stack, the Deployment provider will be CloudFormation. This CloudFormation stack will create a simple EC2 instance in the us-west-2 region. Again, please note that the deployment stage can be changed as per your requirement.
PARAMETERS USED FOR THE CLOUDFORMATION STACK:
PipelineName : Name of the Pipeline to be created. Make sure you do not have a Pipeline that already exists with the same name.
S3Bucket : Name of the S3 bucket where you will be storing the source code Zip File.
SourceS3Key : If you are adding the Zip file to the root of the S3 bucket, you can just mention name of the Zip file (CodepipelineCrossRegion.zip). However, if you are adding it inside a folder in the S3 bucket , you will have to provide the complete path.
For example if you have added the CodepipelineCrossRegion.zip to a folder called CodePipeline in the S3 bucket, enter the value for this parameter as CodePipeline/CodepipelineCrossRegion.zip .
TemplateFileName : Name of the template file in the source code (it can be a JSON or YAML file). For example sampleEC2.yaml.
TestStackConfig : The name of the file containing the information about the parameters to be passed to the cross region stack. For example parameter-configuration.json.
StackName : Name of the Cross Region stack that will be deployed using CodePipeline.
ArtifactStoreBucketTargetRegion : Name of the bucket in the target region where the artifacts will be stored by CodePipeline.
RESOURCES CREATED BY THE CLOUDFORMATION STACK:
ArtifactStoreBucket
Type → AWS::S3::Bucket
This is an S3 bucket that will be created to store the artifacts in the region where the pipeline will be created. It is mandatory to enable versioning for this bucket.
CloudFormation will assign a name to this bucket and you can find it in the Outputs Section.
CrossRegionPipeline
Type → AWS::CodePipeline::Pipeline
This the Pipeline that we will be creating to deploy the cross region CloudFormation stack with the source stage as S3. Some important points :
1. Make sure you are passing the CodePipeline Service role with required permissions in the template. I will be discussing the required permissions later.
2. Ensure that you are passing the correct S3 bucket name and especially the S3 key (path to the source code file in the S3 bucket).
3. In the Deploy stage, it is mandatory to provide the CloudFormation Service role. The permissions of this role is used to create the cross region CloudFormation stack.
4. Confirm if the template file (TemplateFileName) and parameters file (TestStackConfig) are being passed rightly during the stack creation as parameters.
5. The most important thing is to specify the ‘Region’ property in the template and pass the correct region in the parameters (TargetRegion) so that stack is deployed in the required region.
6. In the ‘StackName’ property, you can even specify an already existing CloudFormation stack in order to manage it via AWS CodePipeline. If a stack with that particular name does not exist a new CloudFormation stack is created.
7. In the ‘ArtifactStores’ property , you will have to add an S3 bucket which is present in the target region i.e the region where you are deploying the cross region CloudFormation stack. The behavior through CloudFormation is different from Console.
When you implement this through the AWS console, CodePipeline automatically creates a S3 bucket in the target region but it is not the case with CloudFormation. If you fail to specify this , then the stack creation will fail with an error that looks something like this :
— — — — — — — — — -
Your pipeline must have an artifact store, such as an artifact bucket, for each region where you have an action. The following region is missing in ‘pipeline.artifactStores’: us-west-2.
— — — — — — — — — -
PipelineRole
Type → AWS::IAM::Role
This the service role to be used by the CodePipeline service. This role will need full permissions for S3 and selected permissions for CloudFormation as below :
NOTE: Permissions for this role should be changed if you are adding or modifying any of the stages being created and these permissions depend on the addition/modification. For example if you are deploying to Elastic Container Service , then permissions for that service are necessary.
CFNRole
Type → AWS::IAM::Role
This is the service role for CloudFormation. AWS CloudFormation uses the permissions of this role to create the resources in the Cross Region CloudFormation stack. Hence, this role requires S3 permissions and EC2 permissions.
S3 permissions are required to fetch the template and parameter file from the S3 bucket and EC2 permissions to create and manage the EC2 instance if any update is required in the future. I have added below two managed policies to this role :
NOTE: The permissions for this role should be modified if you are creating different resources in the cross region CloudFormation stack.
Template used to create the EC2 instance using the cross region CloudFormation stack
Source Code for this article available here — SOURCE CODE
DEPLOYMENT STEPS :
Please follow the below steps in order to create the Pipeline that will deploy/manage the cross region CloudFormation stack :
STEP 1 → Create the source code. The source code Zip file (CodepipelineCrossRegion.zip) must contain the YAML or JSON template file and a parameter configuration file.
STEP 2 → Upload the Source code to the S3 bucket in the region where you will be creating the Pipeline. Make note of the name of this S3 bucket since it will be used as the ‘S3Bucket’ parameter during the creation of the main stack.
STEP 3 → Make sure you have a bucket created in the target region to be mentioned as one of the artifact stores. This bucket name should be passed as the ‘ArtifactStoreBucketTargetRegion’ parameter.
STEP 4 → Open the AWS Console and Navigate to CloudFormation console in the region where you would like to create the Pipeline.
STEP 5 → Click on Create Stack option >> With new resources (standard) .
STEP 6 → Choose ‘Template is Ready’ and for the template source , click on ‘Upload a template’. You will get an option to choose the template from the local file. Download the template file or copy the template to a different file and upload it from the local machine. This template file will be automatically uploaded to the S3 bucket by CloudFormation.
STEP 7 → Click on Next. Now enter the Stack Name and the Parameters required for the stack creation. Please refer to Parameters section above for more information on what each parameter is used for in the template.
STEP 8 → Click on Next. Add any Tags to the stack or specify the Service Role if you would like to in this page and also you can make use of other Advanced options provided by CloudFormation.
STEP 9 → Click on Next. Verify if all the parameters have been passed correctly and also check other stack options if you have set any.
STEP 10 → Make sure you acknowledge this message — ‘I acknowledge that AWS CloudFormation might create IAM resources.’ and then click on ‘Create’.
The Stack will be deployed successfully and you can find the following information from the Output tab in CloudFormation console:
ArtifactS3Bucket : The S3 bucket where you can find the Source Artifacts.
CodePipelineName: Name of the Pipeline created in AWS CodePipeline.
CodePipelineServiceRole: Name of the CodePipeline Service Role.
CloudFormationServiceRole: Name of the CloudFormation service used associated with the cross stack.
CONCLUSION
This article will help anyone who is looking to implement cross region CodePipeline deployment using CloudFormation. The example that is shared here is only a basic implementation. By following the same approach and with slight modifications you can create multiple stages with multiple deployments to multiple resources across different regions using CodePipeline that is managed by CloudFormation.
I hope this article was informative and has helped you achieve your final goal . Happy Cloud Computing !!!
