Enhancing HumanGov Deployment with AWS CI/CD Pipelines
Hello Medium Readers,
Last week, as I was deploying the next phase of the HumanGov application, which I’ve been documenting here on Medium since last year, I could barely contain my excitement to share with you what I was working on. Over the past week, I’ve been diving into the world of DevOps best practices, particularly focusing on CI/CD (Continuous Integration/Continuous Delivery) — a fundamental concept in modern software development.
While I won’t delve too deeply into the intricacies of CI/CD pipelines due to my novice level in this area, I have already begun implementing some of these practices in my daily work as a Data Scientist. However, in today’s post, I’ll walk you through how I deployed the latest version of the application using a CI/CD pipeline in AWS. Specifically, I utilized two newer services: AWS CodePipeline and CodeBuild. Throughout this text, I’ll provide a detailed exploration of these services, so let’s get started.
It’s worth noting that while this post isn’t primarily about Jenkins, I did begin my CI/CD journey by learning about it and conducting my first hands-on experience with the tool on an AWS EC2 instance. I set up Jenkins to run on this instance, configuring its pipeline and stages. For those interested, you can find more information and a hands-on guide about this tool here.
Task of the day
As I’ve previously discussed in earlier posts, one, two, three, in my role as a DevOps Engineer, I’ve delved into automating cloud-based services, particularly focusing on optimizing the processes for the HumanGov SaaS application. The goal was to streamline both the build and deployment procedures using Kubernetes while implementing continuous integration and delivery through AWS services.
In this new phase, I’ve incorporated two additional services:
- AWS CodePipeline: This service functions similarly to the Pipeline mode in Jenkins, serving as the platform where you configure your entire CI/CD Pipeline.
- AWS CodeBuild: Acting as a complementary service to CodePipeline, CodeBuild is akin to configuring stages in Jenkins. Essentially, for each step within CodePipeline, a corresponding “Stage” project needs to be set up in CodeBuild. I’ll delve deeper into this aspect shortly.
The proposed solution architecture closely resembles that of the previous post, where I deployed the application using EKS, ALB, Route53 with Certificate Manager. Therefore, I won’t delve into these components in this post. The new addition here is Continuous Integration and Delivery, triggered whenever changes are made to the source code of the application, which is stored in AWS CodeCommit. CodeCommit is a self-managed AWS service for code repositories, similar to GitHub.
To elaborate, the trigger that initiates the CI process is a new commit made to the master branch of the repository in AWS CodeCommit. While I’m deploying a Docker-based solution, it’s worth noting that I could have opted to set up an additional trigger based on the Docker repository, AWS ECR. However, for the sake of simplicity in this deployment, I chose to follow a more straightforward approach. Nevertheless, it’s important to highlight that the same idea could be implemented using AWS ECR.
Implementation
As mentioned earlier, I won’t be covering the same steps discussed in the previous post regarding EKS. Instead, I’ll provide more details about CodeBuild and CodePipeline.
CodePipeline
So, in the image below, you’ll see the view when you select to create a new pipeline in CodePipeline in AWS. As depicted, in the initial steps, I simply select the pipeline name and click to advance to the next step.
Following the initial step, the second step involves selecting the source origin of changes. In simpler terms, you need to choose what your pipeline will monitor to detect any new modifications, prompting the initiation of the CI/CD pipeline.
As illustrated in the image below, I opt for AWS CodeCommit. This means that whenever a new commit is made in the human-gov-application repository, specifically in the master branch, the pipeline will commence. Moreover, the pipeline initiates automatically due to my selection of Amazon CloudWatch Events, as indicated in the option descriptions.
In the third stage, the real work begins, although I must say, it’s not as daunting as it may seem! Here, we need to choose the Project, which is akin to Jenkins stages, that will run within the pipeline. These stages are constructed using AWS CodeBuild. Naturally, I opted for CodeBuild as the build provider, but it’s worth noting that AWS also provides integration with Jenkins and other similar tools.
How to configure CodeBuild
When you click to create a project in the third stage, a new page will open, and you’ll see the image below on your screen. Similar to the process in Jenkins, you have to choose a computer resource that will physically run your stage (build). My selection was to run it on an EC2 instance instead of a Lambda. Here, I selected the operating system, runtime, and image, as illustrated in the image below.
Following that, I enabled the privileged flag to facilitate authentication. The next part is quite important; you really need to pay attention to configuring the network settings. As a first-timer, I opted for the easiest path, using my default VPC and selecting private subnets available within those created by EKS to host the application. The security group was also the same one created by EKS. Towards the end, AWS provides you with the option to validate your VPC settings; you simply need to click and wait for the response, as shown in the image below.
The final step in creating a CodeBuild project is to select the size of the compute resource and define any special environment variables. Typically, you can pass AWS credentials or, in my case for this initial build, the public Elastic Container Service (ECS). Why ECS? Because I want every run of this CI/CD pipeline to generate a new version of the container image in ECS.
Towards the end of the image, you can see the Buildspec, which is, again, similar to the SCM of the Jenkins pipeline. This Buildspec is where I can include all the commands that need to be executed by the stage in the Pipeline. This includes actions such as pushing and pulling container images, creating a new service, updating an existing service, and so forth.
Well, that concludes the CodeBuild project creation process. You can click to save and finish, and then, when a message appears on the screen, you can close it, and the project will be attached to the actual pipeline selection in step 3.
Get it back to AWS CodePipeline
Before starting the fourth stage, at the end of the third stage, AWS will ask you if you want to build a single pipeline or a batch, meaning if you want to have more than one stage. I proceeded with just one, and once I finished this setup, I tested it, and then I added another stage to the pipeline.
In the fourth stage, you can select the deployment provider. I simply skipped this and deployed the stage, while the fifth and final stage is just a review.
After completing all of this configuration, I began adding additional stages to my CI/CD pipeline, 6 at total, with manual authorization at the end. The result of my pipeline is shown in the image below.
Explaining the stages:
- Source: This stage evaluates if a new commit was made in the master branch repository.
- Build: This stage utilizes the Buildspec to push a new Docker image into ECS.
- HumanGovTest: This stage runs tests to check if the new version passes the test — similar to Q&A.
- HumanGovDeployToStaging: This stage deploys the application to a staging environment for testing before final approval.
- ManualApproval: This stage waits for someone’s approval after a careful evaluation of the staging environment.
- HumanGovDeployToProduction: Following successful testing in the staging environment, this stage moves the tested and evaluated version from the staging environment into production.
Conclusion
In conclusion, implementing a CI/CD pipeline in AWS has improved the development process and enhanced the reliability and efficiency of my deployments. Through the stages of source evaluation, building and testing, deployment to staging, manual approval, and finally deployment to production, we’ve established a robust workflow that ensures the quality and integrity of the HumanGov application.
I hope you enjoyed the introduction to these two newer services from AWS. Follow me to stay updated on the next post.
