Universal CICD pipeline for K8S on AWS
At an abstract level, a deployment pipeline is an automated manifestation of your process for getting software from version control into the hands of your users.
The term CI/CD is becoming widely used in the industry right now. You might have heard about it as Continuous Integration, Continuous Delivery or Continuous Deployment. It’s a concept that goes by many names but covers some basic ideas. Briefly, it outlines practices to keep your code more maintainable, testable and get to your users more quickly. Ultimately, it generates value for the user and the consumer.
At first glance, it might seem best suited for enterprises or some capital venture. But by understanding its value, you will find CICD to be valuable whether you’re building scalable enterprise applications or simply getting your personal project up and running.
Decoupling Infrastructure
Develop systems, not software
Microservices has been a widespread term in our industry for a while. It has many shapes and definitions depending on who you ask or the level of detail you want to discuss. But we all agree on one thing: smart decoupling of your applications is the way to go. Whether it is based on the bounded context or application types, you have criteria by which you decouple services in your project.
Hence, it is a good idea to consider this early on. Many developers design a certain level of decoupling from the start. This requires some forward thinking, as you need to decide the decoupling criteria and the tools you will use for this scope. These tools represent the infrastructure (AMQP, Databases, storage and so on).
I could go on for a lot more than this article about decoupling techniques and tools. But here, I will mention the tools I use for this example and get back to the CICD pipeline.
Entering AWS and K8S
A general (and quick way) to maintain infrastructure is through a Cloud System, because of the abstraction and sheer amount of tools they provide. I am using AWS for the variety of its tools. Specifically:
- CodePipeline for CICD pipeline construction
- CodeBuild for running build commands during the pipeline steps
- ECR (Elastic Container Registry) as a repository for build result images.
- S3 for storage
Kubernetes (K8S) is a Docker containers orchestrator. It applies the IaC (Infrastructure as Code) pattern to provide further abstraction. By applying YAML format files to K8S we can update its state, deploying containers or other objects.
I will use K8S hosted on AWS. This can be done using kops. Here is a nice article explaining how-to.
Universal CICD
Yes, great. But where does CICD fit in all this? Well, I started by explaining decoupling to lay down the idea of requiring several services divided by context. The services we develop might have different deployment requirements. Running environments may differ by endpoints, dependencies and so on.
For this reason, we normally have a CICD pipeline for each project.
The abstraction of the infrastructure (provided by K8S) is very useful here, as it allows us to set the environment requirements in a YAML file on the codebase.
Continuing this concept we can abstract the CICD pipeline itself. Making it agnostic of project type, build and deployment requirements.
By having these 3 components in our solution:
- Build commands file. (Buildspec.yml for AWS CodeBuild)
- Dockerfile image of the running application
- K8S object YAML files, for deployment requirements.
We have all the information on how to build, test and deploy any of our services.
CICD schema presentation
The most powerful tool we have as developers is automation
The pipeline must be triggered on every commit. It should pull the source files without using specific information of any repository.
To achieve this, the pipeline will be triggered by a GitHub webhook. This allows for multiple uses by different projects.
Through API Gateway, the webhook will call an AWS Lambda function. This function will pull the git repository and zip it to an S3 bucket.
AWS CodePipeline can be triggered whenever an S3 object is updated. It “watches” for our zip file, and whenever it is updated triggers the pipeline.
CodeBuild uses the buildspec.yml file from the solution, to build and run tests.
Next, a Docker image will be built and pushed to AWS ECR.
Finally, the YML files on the solution will be applied to the existing K8S clusters on AWS. Usually, this results in pulling the Docker image from ECR and deploying it to a container in K8S.
To recap, the pipeline can build and deploy any project as long as it provides
- Buildspec.yml file with build environment requisites and commands
- A DockerFile image in order to push it to ECR
- Kubernetes deployment files under “/k8s” folder
So let’s take this step by step. I use a SCALA PLAY project as an example. You can find the project with the required files here.
AWS Lambda and API Gateway
We can expose an endpoint for an AWS Lambda function using API Gateway. Set this endpoint as a webhook on our GitHub project.
The Lambda function finds the GitHub URL on the json file sent by Github, pulls the solution to a temporary directory, creates a zip and uploads it to an S3 bucket. This will be triggered on every commit.
The resulting zip will be updated to an S3 bucket, which will trigger the pipeline.
On AWS Lambda I use NodeJs version 8.10.
You can find the Lambda function here, written in NodeJs.
Many thanks to Enki for this hack using lambda-git.
The Pipeline
Here is an overview of the pipeline in AWS CodePipeline.
The first step is triggered by an upload to S3 of a specific object. This object will be updated by the previous lambda function. This step generates the “Source Artifact”, which is a .zip with the source files.
The “Source Artifact” is sent to the Build step. The solution is compiled, tested, and generates the build result files as “Build Artifact”.
This artifact is sent to the Push-ECR step. The Dockerfile is built using the “Build Artifact” files and pushed to ECR.
Finally, the Apply-KOPS step applies the YAML files found on the “Source Artifact” on K8S with kubectl. Additionally, it saves the K8S YML files to an S3 bucket. This stores information about the current infrastructure.
Continuous Integration
The Build Step is an AWS Code Build project outlined here. To configure a CodeBuild project, we specify a build environment and build commands.
The build environment is the CodeBuild standard 1.0 -1.8.0.
Under the hood, AWS CodeBuld creates these Docker images as build environments. For more info, you can check these images here.
This project will get the Buildspec.yml file from the source. The YAML file contains the build commands to execute.
Here is a buildspec file for a project in SCALA PLAY.
The project will start the compilation and run tests. If everything is green, a zip file will be created as a «Build Artifact». It will contain all the necessary files to run the application.
Push Docker Image
On this stage, a Docker image is created and pushed to ECR. This is done by a CodeBuild project as well.
We insert the buildspec commands directly into the project. These commands expect a Dockerfile on the source root. This requires the «Build Artifact» files to create an image with the running application. The image is built and pushed to AWS ECR.
The Dockerfile image runs the application, assuming it was built.
Here is a Dockerfile to run a project in the PLAY framework in SCALA. It unzips the build result and runs it using the provided secret key.
The Build environment is the same standard 1.0 -1.8.0.
Deploy to K8S
The assumption is that all YML files under the “k8s” folder are Kubernetes objects (generally deployments and services). These files are pulled and applied to the Kubernetes cluster. In order to do this, we need kubectl. I have used this Docker image as a custom build environment for CodeBuild.
Push this image to an ECR repository, and set it as build environment.
The buildspec for this project is the following
All K8S objects under folder “./k8s/” are applied and then saved to an S3 bucket, to store the current cluster state for future reference. Kubectl obtains the K8S cluster configuration using ‘export kubecfg’ command.
For example, these are the K8S objects for the project in SCALA PLAY.
Through K8S, this will result in deploying a Docker container. Also, AWS will create a Load Balancer pointing to the container.
Conclusion
Being able to deploy a service solely by its code is a powerful tool for a developer. Through this, you can work on several services and deploy their infrastructure in a decoupled pattern. All while maintaining CICD advantages without having to worry about its specific implementation.
The next steps towards abstraction may be using bash commands on the solution instead of Buildspec, which is specific to CodeBuild. Making it compatible also with Jenkins or other build tools out there.
