Hands-on Day 1 and Day 2 Operations in Kubernetes using Django and AKS — Part 4

Kubernetes has become the de facto container orchestrator due to its various functionalities and flexibility. Although Kubernetes documentation is thorough and provides many examples, it is not straight forward to combine all these tutorials and use it to deploy a real-life application with several services from end-to-end. With that, I will demonstrate how to deploy a real-life scenario application on Azure Kubernetes Cluster (AKS) as the production platform. Moreover, I will be discussing day 1 and day 2 operations of the application life cycle in Kubernetes in series of articles. Here are the main headlines:

• Discuss the application and set up the cluster, container registry and the production namespace. (part 1)
• Deploy Config Maps, Secrets and Persistent Volumes. (part 2)
• Deploy, monitor and define update strategies for the services including setting up Traefik as Ingress Controller. (part 3)
• DevOps and Auto deployment using Github Action. (part 4)

You don’t have to use Azure Kubernetes Service per say, you can easily re-configure the manifests to be compatible to any Kubernetes installation, such as AWS EKS or Linode. However, as a prerequisite, you need to have a basic knowledge of Kubernetes, Docker, yaml, shell scripting and Github Actions. In addition, if you want to run the application along with the tutorial, you need to:

• Clone the Cookiecutter-django-k8s repository.
• Install Docker and Docker-compose.
• Enable or install Kubectl.
• Install Azure cli.

In this article, best practices for continuously deploying an application in Kubernetes cluster is discussed. Then, a demonstration is provided on how to implement a Continuous Deployment pipeline for the presented application in part 1 using Github and Azure Actions.

Continuous Deployment

A key benefit of implementing a Continuous Deployment (CD) pipeline is to automate the release of the software in an efficient, fast and trackable way. In theory, Kubernetes makes CD easy to implement due to its declarative nature. Moreover, you describe the state that you want the applications to be. Then, Kubernetes attempts to reach that state. However, as we saw from the previous articles, Kubernetes takes care of the complete infrastructure; starting from setting up the applications’ configurations, jobs and disks to deploying the applications themselves. Monitoring the changes of these different components and automate the CD become very challenging. In other words, the more different Kubernetes resources you use, the harder it is to automate the complete pipeline.

Let’s briefly describe how ideally we can implement CD pipeline for the main Kubernetes resources used in the previous articles:

ConfigMaps and Secrets

In some cases, you end up updating only the application’s configurations, i.e. ConfigMaps and Secrets, without updating the microservices themselves. When CD is ran, the configuration changes are not reflected in the microservices unless you restart the microservices that are using them. This happens because microservices and configurations have different lifecycles and constraints.

One way to solve this issue is within the logic of CD pipeline. Moreover, we look for changed configuration manifests: if there is any, we will force restart the microservices that are using these configurations unless these microservices are modified anyway. You can simply check whether a configuration is modified or not by using the option --dry-run that will inform you the output of the command without actually applying the changes. Another method is to check if the last changed files in the repository contains any ConfigMaps/Secrets manifests using for example verify-changed-files Action in your Github Actions workflow.

Although the idea is simple, the implementation is complicated and error-prone especially when a configuration is used by multiple applications across multiple teams. A better practice is to treat each configuration file as immutable file where you need to create a new version of the ConfigMap/Secret if you want to change its values. With that, you will need to modify the microservices manifests to load or mount the configurations of the newer versions. This gives you the possibility to let other applications that depend on the same configuration to use the current version of the configurations. This approach is followed by Spinnaker.io.

Persistent Volumes

In Persistent Volumes, one normally wants to expand the size of an already created volume. This can be easily achieved by modifying the Persistent Volume Claim (PVC). Specifically, the resources.requests.storage attribute. Unlike ConfigMaps/Secrets, updating the PVC will be reflected directly without requiring us to restart the related microservices.

However, if any of the volume type or class storage are required to be changed, then it is very difficult to handle. In this case, a good practice is to re-create the related microservices again with the new version of the PVC, this approach can be better managed using Infrastructure as Code (IaC) tools such as Terraform. IaC tools require different handling in CD that is out of the scope of this article.

Microservices

The microservices that define the applications are the most common resources to be updated. Essentially, we would like to reflect the changes and the new features in production whenever we make a release. This can be easily done by compiling new docker images that contain the latest code and push these images to a container registry. These images are usually tagged with the release version. Then in CD pipeline, the services are forced to be updated by pulling the images from the container registry with the correct tag. Normally, the services are updated along with their correspondent Kubernetes Service.

The update strategy is defined using rolling update configurations and health check probes that have been explained in part 3. Additionally, we can define further update strategies in the CD pipeline such as Canary strategy or Blue-Green Strategy which I will demonstrate how later. If you are unfamiliar with these mentioned strategies and you would like to learn more, then please refer to this thorough tutorial.

Demonstration

Let’s demonstrate the CD pipeline using the same example repository and Kubernetes cluster from the previous articles. The repository’s complete CI/CD pipeline is split into three different stages:

Three Stages of CI/CD

In a nutshell, the pipeline starts when a pull-request is submitted, the code style will be linted and tests will be ran. Once merged to master, docker images will be compiled and pushed to the specified container registry. Finally, upon release, we tag the compiled images from the last step with the release version and update the Kubernetes production cluster. Each stage is split into different Github Action scripts in the repository. In this article, I will only discuss the last two stages that are related to CD as the CI stage was already provided by Django Cookiecutter.

Environment Variables and Secrets

Before we start with Github Action scripts, we need to provide Github the necessary environment variables and secrets that are required to run the pipeline. Mainly we need to set:

• Azure’s access information.
• AKS cluster and Resource-Group names.
• Azure Container Registry (ACR) credentials.

In Github, we can set the environment variables as secrets in settings -> secrets. However, it is good practice to group secrets that are related to an environment in Github, this helps us in the following:

• Assigning different people in the team for different stage of the application’s lifecycle. For example, allow specific members to create a release.
• Re-use the jobs in Github Actions and only change the environment. In most cases, deploying Kubernetes to staging cluster is the same for production cluster. the difference resides in the access and credentials information. With that, we can make the job general and specify to run it on different environments.

With that, first we create an environment called productionand assign the following keys to it:

• AZURE_ACCESS_INFO
• CLUSTER_NAME
• RESOURCE_GROUP

AZURE_ACCESS_INFO can be retrieved by running the following command in Azure cli (you only need to run this command once):

az ad sp create-for-rbac --name "aksaccess" --role contributor --scopes /subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP> --sdk-auth

The command will return a JSON object that you can copy and store as the value of the AZURE_ACCESS_INFO variable. The other two secrets are self-explanatory and you should have obtained them when you created the cluster (for further information check part 1).

Environment variables and secrets that are unrelated to any environment can be stored in the general Github Secrets. For example, the container registry information are mostly shared between production and staging environments and therefore unrelated to a specific environment. With that, navigate to settings -> secrets and add the following variables:

• CR_URL
• REGISTRY_USERNAME
• REGISTRY_PASSWORD

You can find these information in ACR dashboard in Azure’s portal under Access Keys tab. (For further information refer to part 1).

Github Action Workflow

Let’s start first by compiling and pushing the images to ACR. Here are the main steps to be followed:

1. Checkout the code.
2. Login to Azure docker.
3. Build the docker images locally.
4. Iterate through all images in docker-compose file and push each image.

The following snippet shows how to implement the steps in Github Actions:

Snippet on pushing docker images to ACR

First of all, The job, push, is only triggered when a commit is pushed to the master branch as stated in line 1 to 4. The steps are straightforward. However, the final step in line 18, is ran within shell script named push-images.sh. The following snippet shows the shell script’s implementation:

Shell script to loop through the images from the docker-compose file, tag and push the images.

Basically, the shell script accepts an argument to set the images tag, if no argument is passed, then latest is chosen. Afterwards, it gets the configuration from the docker-compose file, production.yml, and only tries to find image attribute. Finally, it loops through all images and for each image, it tags the image and pushes it to the container registry.

Once we have a stable version of our application, we create a release. Beside production release, a release can be alpha, beta or a release candidate (rc*). Normally, you would like to trigger different environment for each release type. In this simple example, only production release is considered which normally starts with v*. like for example, v1.0.0. With that, we trigger the CD pipeline when a release is made that starts with v* as shown in the snippet below:

Set how the CD workflow will be triggered.

Then, we need to tag the compiled images with the release version. The following snippet illustrates Github Action script for tagging and versioning the docker images:

Get release’s version, tag and push the images.

The steps are very similar to the previous job, push , the only difference is that we pass a parameter to push-images.sh script. The parameter is defined in the Get Release Version in line 16, where we retrieve the release version from Github’s internal environment variable, GITHUB_REF . Finally, for performance purposes, we pull the latest images that are built from the previous stage instead of re-building them again.

Once the job is done, we can deploy the changes to AKS’s Cluster. Here are the steps that are required to deploy to AKS:

1. Checkout the new code.
2. Login to AKS.
3. Get release version
4. Get list of Kubernetes manifests.
5. Get list of images.
6. Deploy to cluster.

The following snippet illustrates how the steps are implemented:

Job to deploy to AKS cluster

First of all, in line 3, we define that this job runs with respect to production environment that we created earlier. In step 2, Azure’s action aks-set-context is used to programmatically login to AKS, the action depends on the secrets we defined in production environment. In addition, Azure action k8s-deploy in line 33 is used to deploy to cluster. It expects list of manifests to run, the new compiled images’ URLs and the targeted Kubernetes Namespace within our cluster. The list of manifests are retrieved in step 4 using shell script list-manifests.sh . The script simply loops through the Kubernetes folders and gets the path to each manifest. Here is the how list-manifests.sh is implemented:

Shell script to list the Kubernetes Manifests to be updated in the cluster.

The order of the manifests is important, as you want to apply changes of ConfigMaps and Secrets first before you update the microservices.

Finally, the list of images are retrieved using list-images.sh that is very similar to push-images.sh except it only gets the images URLs without pushing them to ACR.

k8s-deploy can take further argument to define the deployment strategy, it accepts both Canary and Blue-Green strategies. For that, you are required to provide additional variables for Canary strategy such as the percentage of replicas of baseline and canary variants. However, you don’t need to modify your Kubernetes infrastructure or introduce additional resources.

Finally, if you didn’t link the AKS cluster with ACR when you created the AKS cluster (as we did in part 1), then you will be required to provide the access to ACR through parameter in k8s-deploy action. Further information can be found in this tutorial.

With that, we have a CD pipeline using Github Actions to roll updates of the newly released features to the production Kubernetes cluster.

Things To Consider

• Use Helm to automate the deployment of Kubernetes and make the manifests more customizable.
• Use Terraform to automate the infrastructure changes.

Clean Up

If you ran the tutorial, please go ahead and delete the resource group, Azure then will delete every resource in that group. Additionally, delete the service principal that has been created along with the resource group.