API services on cloud #3: CI/CD integration
Note: this is the Part 3 of the “API services on cloud” series on running API Services on cloud and learning cloud infrastructure automation. Stay in the know. There’s more of this to come!
< Previous | << Start over at Part 1
Intro
In the previous two parts, I described how to create a minimalistic infrastructure to run simple API service on AWS. The deployment process was either completely manual (via AWS web console) or via the scripts. For the ‘hello world’ like projects, such approach is more than enough. But the real world projects becoming complex fast. And it’s important to set the build pipelines as early as possible. That is why the subject of this article is CI/CD integration.
AWS User & policies for CI/CD
Previously I was deploying infrastructure with the help of the bash scripts using my own AWS profile. This user has lots of privileges and it’s not secure to use it in CI/CD environment. Moreover, I prefer to create a user per task. Such users have limited permissions which required only for the task execution.
Thus I created a user, especially for CI/CD. Its profile will have permission only on those operations which required for the Simple-API stack deployment. Bellow, you can see the body of the simple-api-cicd-policy.
S3 permissions required to upload and read CloudFormation templates from a bucket. Cloudformation permissions required to create or update stack. EC2, autoscaling, and elasticloadbalancing permissions required to support autoscaling and create the load balancer. Cloudwatch and logs permissions are responsible for the creation of the log groups and streams. IAM permissions allow CI/CD user to create instance role.
Permissions on delete operations required for two reasons. First, when stack creation failed CloudFormation tries to delete created resources. If CI/CD user does not have delete permissions then the cleanup process stuck, the stack cannot be deleted, and the resources hang. Second, stack update operations might require replacement of the existing resources.
To create a policy navigate to IAM > Policies in AWS web console. Then click ‘Create policy’ button.
On the first step choose JSON tab and paste policy body provided earlier.
On the next step review the policy and click ‘Create policy’ button.
After the CI/CD user policy has been created we need to create a user itself. To do it navigate to IAM > Users in AWS web console. Then click ‘Add User’ button and follow the user creation process.
Note, for CI/CD purposes only programatic access required.
On the last step, you will be provided with the user’s credentials. You can download credentials.csv and securely store it. Later, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY will need to be injected to CI/CD.
CI/CD integration
When I was deploying stack manually I could verify if it’s already deployed to make a decision what script to run: create or update stack. But now this decision should be done automatically. Thus I decided to rewrite my deployment scripts in NodeJS.
The script scans the cloudformation directory of the project and uploads all files to the bucket. The bucket passed as command line argument and contains bucket name and directory. This is a reason to make split, shift and join operations on lines 14 - 18.
The deployment script lists existing stacks and makes a decision to create or update Simple-API stack. After the create/update operation has been initialised the script is waiting for the operation to finalise.
The script runs unit tests and then uploads the stack artefacts to the bucket. After the deployment process starts. Note, due to a time required to create the stack (it can be more than 10 minutes) it’s necessary to add travis_wait instruction to prevent CI from stopping. By default, it stops after 10 minutes without a response from the build script.
Also, as you could notice build configuration does not contain any AWS profiles or credentials. To make it work I injected AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables to Travis environment. These variables will be automatically picked by aws-sdk used in the build scripts. Go to the project settings at Travis to add the necessary environment variables.
Versioning
In addition to automation of deployment process, CI/CD provides a versioning capabilities. As you could see in the upload script above - every build uploads template into location related to the build number. Not only the templates could be stored there. Other deployable artefacts like service binaries, static resources can also be stored there.
Thank you
That’s all for today. I thank you for reading till the end. Please ‘clap’ or/and comment if you liked it. If you did not like it, please comment why — so that I will be able to improve it next time.
References
- Simple-API - NodeJS service and CloudFormation source code.
