Trunk-based development Enabled by Kubernetes Accelerator
Containers enable us to build modern applications or package existing applications for increased portability and reduce environment drift. Container orchestrators like Kubernetes allow us to distribute the containers intelligently across our infrastructure, either in the cloud or on-premises. Furthermore, container orchestrators simplify deployment processes, ensure high availability, provide scaling capabilities, and may abstract capabilities from the application itself. This reduced overhead lets developers reclaim time and energy that can be spent on developing great products.
Container Orchestrator
Some of the benefits of a Container orchestrator like Kubernetes are:
· Maximize Compute Infrastructure — Container orchestrators intelligently schedule containers on machines to maximize the utilization of the host.
· Flexible Scaling — Applications can be scaled automatically by adjusting capacity.
· Automated High Availability — Container orchestrators provide high availability and resilience; if a node or host dies, the containers will automatically get rescheduled in another available node.
· Lower Risk Deployments — Not only do containers ensure that the code will work the same way in every environment, container orchestrators allow for rolling zero-downtime deployments.
Why use Kubernetes?
Kubernetes is the leading container orchestrator tool today. Kubernetes exhibits all the great qualities of a good container orchestrator, i.e., intelligent scheduling, self-healing, resilience, load balancing, rolling updates plus rollback, and horizontal scaling. Gartner estimates that by 2022, more than 75% of global organizations will be running containerized applications in production.
Trunk-based development
As more and more teams adopt trunk-based development, we wanted to unlock the potential of using trunk-based development on Kubernetes. We also wanted to reduce risk and increase productivity by making it easier for developers to test their feature branch changes before merging to trunk.
Trunk-based development is a branching model in which all branches extend from a single trunk or master branch and are merged back to the master after getting tested. The benefit of this development model is that it can help seasoned developers move quicker by doing smaller and frequent merges without having to deal with unnecessary bureaucracy. Since merges are applied to the master directly, the code is always production-ready. The flip side is, while this workflow provides a lot of autonomy, there is an inherent risk of introducing immature code or bug if proper code review and adequate testing are not performed. Hence it is imperative to quickly provision and provide every developer with a ‘shared-nothing’ isolated test environment to test each commit to their feature branch.
Slalom Kubernetes Accelerator
Many organizations adopting trunk-based development are faced with challenges like their developers not merging early and often. Getting environments stood up for demonstrating changes is also sometimes a long and arduous process. To solve these challenges, we came up with a simple solution. We provision and configure a Kubernetes cluster such that when a developer creates a new feature branch in the source code repository of an application, we automatically provision a new ephemeral and isolated test environment within the Kubernetes cluster. Then we deploy the feature branch to it to let the developer test out the changes made. When the feature branch eventually gets merged and deleted, we automatically clean up the test environment related to the branch and reclaim any resources allocated to it. We named this solution as the “Kubernetes Accelerator.”
Developer experience
The detailed developer workflow is described below:
1. A developer creates a new feature branch say feature-one in the source code repository
2. The developer makes code changes, commits and pushes the updated code to the new feature branch
3. A git repo webhook triggers an AWS CodeBuild job that builds a docker image from the feature branch and pushes the same to a private ECR docker registry
4. The “deploy” stage of the CI/CD process uses Helm to package and manage the deployment of the application to a new isolated namespace through a Kubernetes deployment using the freshly minted Docker image. Helm creates a new Kubernetes service to expose the application and an ingress object is provisioned within the namespace to allow incoming requests. Kubernetes network policies are created to enforce namespace boundaries.
5. The deployment takes 2–5 minutes after which the developer can access the application at https://www.domain.com/feature-one
6. Any subsequent commits to the same branch will just update the underlying docker image of the deployment and not create a whole new environment.
Under the hood
Our Kubernetes cluster runs on Amazon EKS, which is AWS’s managed Kubernetes platform. Amazon EKS makes it easy to run Kubernetes on AWS without the need to provision or maintain our Kubernetes control plane. We use the tool eksctl in an AWS CodeBuild pipeline to provision the Kubernetes cluster with a few other goodies as detailed below:
· NGINX Ingress Controller allows us to expose http and https routes from outside the cluster to applications within the cluster. The Ingress controller uses an external load balancer, and, in our case, we decided to use the Network Load Balancer (NLB). There are many benefits to using an NLB in contrast to an Application Load Balancer (ALB). One advantage is that a single NLB can be shared across feature branches, unlike that of an ALB, which needs to be provisioned separately for every feature branch and that can get pretty costly if you have a large number of feature branches. Refer to this link to read through all the other advantages of using an NLB over ALB — https://aws.amazon.com/blogs/opensource/network-load-balancer-nginx-ingress-controller-eks/
· ExternalDNS helps automatically synchronize our external DNS provider, which is Route53, with exposed Kubernetes services and ingress objects running within the EKS cluster.
· Calico Network Policy Engine helps us isolate traffic within and across namespaces through network policies to secure the test environments.
· EKS CloudWatch Container Insights collects, aggregates, and summarizes operational metrics and logs from the deployed applications
· IAM Roles for Service Accounts (IRSC) to enforce fine-grained IAM roles for Service Accounts running Pods, which is a security best practice.
Trivia
· We use a TLS certificate provisioned through the AWS Certificate manager on the listener of the NLB for our HTTPS traffic. We terminate HTTPS at the NLB instead of terminating it within the cluster to simplify things. An annotation service.beta.kubernetes.io/aws-load-balancer-ssl-cert configured with the ARN of the certificate on the NGINX ingress controller’s service allows it to configure the HTTPS listener on the NLB appropriately.
· We intentionally avoided creating new Route53 records for every new feature-branch because DNS propagation takes several minutes, so instead of using a subdomain per feature with URL such as feature.domain.com, we opted for a path-based URL instead, i.e., domain.com/feature. We set another annotation to rewrite the URL — ingress.kubernetes.io.rewrite-target to “/” in the Ingress objects to keep it transparent to the application.
Clean-up
After a developer merges a feature branch back to the master branch and deletes the feature branch, a nightly job triggered by an AWS CloudWatch rule fires another AWS CodeBuild job that deletes the feature branch’s namespace and any other Kubernetes artifacts related to it. This clean-up job also deletes any docker images of the feature branch from the ECR.
The beginning
We have few other enhancements lined up like the introduction of a Cluster Autoscaler to add/ remove additional worker Nodes dynamically, log aggregation using Fluentd, Elasticsearch and Kibana, securing secrets with SealedSecrets and a service mesh like Istio or AWS App Mesh. Based on some recent Customer interests, there are also plans to port the accelerator solution to other flavors of Kubernetes!
Written by: Dhiman Halder, with inputs from Jon Perkins, Matt Mousseau, Edgar Sanchez, RJ Jafarkhani, Todd Klitzner, and David Smith.
Found this article useful? Hit that clap button. Really like it? Hold the clap, give it two, or fifty! Follow Slalom Technology and read more articles on thought leadership in Technology.
