Kubernetes on OpenStack
At THG there are a variety of applications and micro-services. Some of these are more suited to run on virtual machines whilst some (particularly RESTful micro-services) are very suited to running in a container.
The long and winding road
In the THG WMS team, the last year has been a year of technology migration from public cloud to private/hybrid cloud.
The large-scale technology shift that the team undertook throughout 2019, just laid the foundations for further refactoring and technology updates. The initial focus was reducing the amount of database technologies in use to simplify operations and support (see the Consolidating Database Technologies post).
The second, a large-scale refactoring of the source code to reduce coupling between components, simplifying the codebase, updating dependencies and strengthening the foundations upon which the business logic ultimately rests (in The Big Refactor post).
All this “non-functional” work was delivered alongside features and new warehouse openings in 2019 and into 2020. The team were keen to capitalise on this refactoring work and to start converting some of the smaller components in the stack into containerised micro-services.
Containers & Docker
Some components were already containerised for testing and development purposes; however, moving to running containerised applications in production had been side-lined during the large-scale cloud migration. Now that this migration had been completed, it was time to pick-up the containerisation work again.
The WMS technology stack is a mix of Java & Scala components, along with message queues for events. Not all of these components would be a good fit for a container as a micro-service (for example, some components consumed a large amount of ram and cpu resources and it could be argued these would be better off still deployed directly to a virtual machine).
For those components or micro-services that could be more easily containerised, we needed a resilient and production-ready container orchestration platform: Kubernetes.
Kubernetes
As container technology became more prevalent, there was a tussle around which container orchestration platform or technology would be the best way of managing containerised workloads. Now, nearly 7 years after Docker was first released, Kubernetes has come to the forefront as the de-facto standard for orchestrating container workloads.
Kubernetes may have won the mind-share for orchestration, it is however a challenging and complex software suite to configure and run on your own hardware. To address these challenges, a variety of offerings have sprung up — from hosted solutions like EKS, AKS and GKE to simplified tooling to help install kubernetes.
Kubespray & Kops
Two of the most mature projects designed to make kubernetes installations easier to manage are kubespray and Kops. Kubespray is based on a large set of ansible playbooks, roles and scripts, kops on the other hand is native go code that interfaces with the kubernetes APIs and with the APIs of the underlying virtualisation technology (eg. GCE, AWS or OpenStack).
The advantage of kubespray is the sheer flexibility due to the huge amount of support it has from the community. Nearly all underlying platforms are supported including a bare-metal approach. The big downside to kubespray is performance — running ansible to install and configure such a large amount of software takes time.
Kops is more focused, targets fewer environments and is significantly faster. However it also has flaws, mainly due to a smaller community and due to the fact that the testing priority is for the large cloud providers (which means that the OpenStack provider — the integration we were most interested in — takes a back seat).
Evaluation
From the available documentation, both of these seemed to provide a solution to installing kubernetes on OpenStack, so it was time to trial each approach and decide which solution was the best fit for us.
Looking at Kops first, initially this looked like it would be a good fit for our current needs. Faster than running a large bundle of ansible and a native integration with OpenStack through the gophercloud API, it is reasonably documented and operationally simple. We had previous experience using Kops with AWS & EC2 so knew what it was like to work with.
Unfortunately we uncovered some short-comings of the current integration with OpenStack, even after submitting patches to fix some of the problems, it wasn’t quite ready.
Over to kubespray then. Our experience with kubespray was better and we managed to use kubespray to create a valid kubernetes cluster on both bare metal and on OpenStack. However, kubespray brings a *lot* of baggage with it, including support for any number of public cloud providers that are of no interest to us now.
At this point we discussed moving away from the two main “automated” processes (Kops & Kubespray) and instead, investigated how much work it would be to automate the standard cluster setup using Kubeadm.
For most of the instructions for kubeadm, it’s simple to automate using ansible tasks, the only complexity seems to be around how to handle adding the 2nd and 3rd controller nodes to the classic 3 node control plane. We’ve worked with automating other clustered software(CockroachDB) in the past:
The solution we used was based on creating a lock in consul to ensure that only one node is performing the first initial steps while the other two nodes wait until those first steps are completed.
Having a technical solution to the cluster setup commands in kubeadm at hand, it made sense to build up an entire kubeadm-based k8s automation solution as it would be much simpler than kubespray and wouldn’t bump into the same roadblocks that Kops is currently suffering from.
Automating Kubeadm
Having decided on a strategy, the goal now was to get a viable k8s control plane and worker nodes up and running on our internal OpenStack environment to allow the engineers an opportunity to test migrating some of their applications from VMs to containers.
Following on with previous experience of writing infrastructure-as-code, we continued using cloud-init & ran local ansible processes documented in a previous post.
HashiCorp’s Terraform is our go-to technology for platform independent infrastructure-as-code. Recently, however, there is a new contender that I wanted to investigate: Pulumi.
Unlike terraform, pulumi supports multiple languages; python, go, javascript and typescript. Since only one of those options supports static-typing, it seemed like the largest departure from the terraform DSL and a good test of pulumi’s credentials.
Pulumi
As a starting point, we kept the essential patterns from previous terraform, cloud-init & ansible combinations and simply replaced the terraform which is the bootstrapping part of creating the resources with pulumi. Pulumi does offer further support for application definitions (including defining kubernetes deployment descriptors using code rather than the cumbersome jinja2 templating favoured by helm).
We need to define some constants for the various dependencies and machine types (flavor in OpenStack terms). One specific area of interest is how to manage different stacks (eg, production, staging, dev). Here we use lookup specific config via pulumi.Config which reads data from a stack settings file.
Async/Await
As Pulumi uses an asynchronous programming model, it is important to note that it isn’t possible to use the os.keymanager.getSecret and similar methods without wrapping the code in async:
k8s-cluster
Inside the anonymous async function we can use standard iteration to create a set of nodes to serve as the control plane for our kubernetes cluster. As part of the configuration for each node, we pass in metadata that is used in the cloud-init stage.
It’s just TypeScript
Unlike terraform, pulumi scripts are standard TypeScript. This means that when needed, we can reach out for standard community libraries for manipulating CIDR or for templating a file using lodash…
Terraform on the other hand doesn’t have the same breadth of libraries available, although it handles templating files and manipulating CIDRs using in-built functions just fine.
Putting these fragments together we get this final pulumi script for the resources (1 bastion host, 3 control plane nodes and 2 workers):
Control Plane Ansible
With the raw resources specified, it’s time to take a look into the kubeadm commands that need to be automated. As the control plane VMs boot, we use cloud-init to initiate the process of running ansible on the fresh VM. A standard script injected in as user-data pulls metadata from the OpenStack metadata service and uses this to determine which ansible playbook to run.
Locking for cluster formation
Just as we looked to consul to provide a locking mechanism for creating a CockroachDB cluster, once again consul is used to allow the creation of all VMs at once while blocking 2 of the 3 control-plane nodes from joining the cluster until the first node has completed the initial setup.
Finally, we run the kubeadm commands to initialise the cluster using the first node and to create the join tokens/hash values for the additional control plane nodes.
The remaining nodes (both control and worker) join the cluster after checking in consul if the cluster has finished initialising, and then we can interact with the cluster via kubectl
Open source
The complete set of pulumi TypeScript, Ansible and supporting files are available at THG’s open source github organisation.
We’re recruiting
Find out about the exciting opportunities at THG here:
