Iterating & Adapting
Our journey with Kubernetes and Cloud
by Tom Burton
An Idea.
Welcome to the inaugural post of the Alteos Technology Blog!
Here we’ll share the technical challenges and discoveries we’ve made along the way.
In 2018 we founded Alteos, with the focus of automating the German insurance market.
To do this, we would have to build an API that was both scalable and configurable.
Modularity would play an important role in the success or failure of this new venture.
Our journey has been one of exploration and discovery. We tried a wealth of technologies and tooling to find the perfect combination for us.
In this first blog post, we will be walking you through our journey with Kubernetes & Cloud, describing on a high level the supportive tooling that we used.
The point is not to explain in detail how to set up these tools yourself;
but rather, to illustrate the decision making behind our choices, aimed at maximising the system efficiency.
So, with that in mind, let’s go back to to the birth of our first staging and production cluster…
Provider.
The cloud market is skyrocketing. Analysts predict that the global market will be worth in the order of $623.3 B by 2023. Analysts estimated that 94% of current enterprises use some kind of cloud service.
We knew it from the very beginning:
if we were going to be fast and scalable, then we had no option but to go cloud native.
We store every resource and file on servers located in Germany.
An important question to ask at an early stage is, which of the hundreds of cloud providers out there should you use?
Due to the complexity, cost, and responsibility of such a business, there are only a few key players:
AWS leads with around a 33% market share as of 2019 and revenues greater than its two main competitors combined.
AWS is also the go-to choice for new cloud customers and beginners with a 53% adoption rate.
However, as the graph below illustrates, companies don’t stick to one cloud provider. There is this concept of multicloud or hybridcloud. Companies will spread resources across multiple cloud providers to reduce risk and spread costs:
We settled on a cloud native solution hosted in AWS, due to the need for ensuring high availability as well as scalability for our API.
Furthermore, AWS has a pretty generous Free Tier allocation during the initial setup, including many of their most popular services. This explains the high adoption by startups and SMEs.
As of March 2020, AWS has made huge upgrades to their console: accessibility has considerably grown for DevOps and Developers alike.
Cluster Bootstrapping.
Cluster management is an area that becomes more accessible every day.
A cluster in essence, is a set of machines and associated resources in which applications can run.
Yet, these associated resources are many in number. It is a feat of mental energy to provision machines, security configs, load balancers etc.
When starting out, we looked for a simple solution to get going as fast as possible. Our team settled on using KOPS (Kubernetes Operations).
A lightweight, simple cli that bootstraps everything you need to set up your cluster. It can have you up and running in a matter of minutes with one simple command:
$ kops create cluster --zones eu-central-1 prod.example.comThis is quite a leap from provisioning the whole ecosystem yourself. In fact, this is how our cluster has remained. We would say if that doesn’t attest the robustness of this software, then we don’t know what will.
In late 2019, we realised we wanted to make a technology switch while preserving our cloud native philosophy. Amazon had released EKS and it looked like the perfect way to go.
Unfortunately, it seemed that they were releasing a new minor version for it every week. We decided to hold off until it was little bit more battle tested. After all, KOPS had served us very well for the last year.
In late 2019, there was another development which made us again revisit EKS. Weaveworks released a beautiful cli for managing EKS clusters. It was as simple as KOPS in its design:
$ eksctl create cluster -f eks/staging-cluster.yamlWith EKS, you can manage the various resources in a simple yaml file. This was a huge step up in configurability from KOPS.
Our first step was moving our staging cluster from KOPS to EKS. It was amazing to see another tool provisioning an entire cluster with one command.
But there was one problem: it was really slow. Typical provisioning took anywhere between 15 to 30 minutes. We knew we could do better.
Near the start of this year, we decided to move to our own managed solution using Terraform, by far one of the most popular IaC (Infrastructure as Code) tools on the market.
What is crazy is, they are not even at v1.0.0 yet! In short, you define all resources as modules which you then plan and apply to your cluster. They support everything from AWS EC2 instances to GCP buckets to Kubernetes namespaces.
We built out our own repository containing all the necessary components to create an EKS cluster. This included VPCs, security groups, EKS configuration etc. We bundled all this together in a directory called `cluster`. Then using Makefile we were able to replicate our own eksctl command:
$ make apply cluster stageAnd voila! We could build an operational cluster in a matter of minutes. This was a far more extensible solution, giving us total control over the entire pipeline.
Yet, this solution came after many iterations and may not be the perfect route for you. Consider what you need and ask yourself whether a managed solution like eksctl/KOPS suits better.
Templating.
To maintain a Kubernetes cluster, you generate files known as manifests. These contain all the necessary information Kubernetes needs to orchestrate your cluster, including resources such as ingresses, secrets and deployments.
Writing up resource definitions for one application is pretty simple. Yet if you’re doing this, we would recommend not using Kubernetes. But if, like us, you are running many services in a single namespace then you should consider using Helm or Kustomize.
Helm.
Helm is by far one of the most popular templating tools out there. It uses this concept of Charts to define the resources for a particular application. For example, imagine in your cluster, two frontends that have respective ingress rules and secrets.
In your chart you can define the resources associated with this application. You would have links to your ingress, secrets and deployment all there. The best part is you can reuse this across namespaces, clusters and applications.
Our initial excitement of this functionality was soon extinguished. In Helm v2 you had to deploy another application to your server called Tiller, which would act as the connection between your client Helm and server Helm. This implies that:
- Tiller will usually needs admin privileges. If a user wants to install a chart that contains any clusterwide element like a ClusterRole or a CustomResourceDefinition (CRD), Tiller should be privileged enough to create or delete those resources.
- Tiller should be accessible to any authenticated user. Any valid user of the cluster may need access to install a chart.
At the time of writing, Helm released version 3. In this release they removed the need to have a Tiller deployment in your cluster. Now Helm looks like an attractive option for us and we are considering moving all our services to charts. For now though, we are using a different tool called Kustomize.
Kustomize.
Open sourced by Google, Kustomize is another popular templating tool. The main concept is that you arrange your cluster directories into “bases” or “overlays”.
A base folder defines configuration that will be common across environments. This includes pod memory allocation and common environment variables. You can think of it as a Helm template only with out all the “if” conditions.
An overlay contains environment specific resources. At Alteos, we use stage, sandbox and production to segregate our environments. In these overlays you will find config specific to the namespace. This includes environment variables, secrets, and image tags.
tree
.
├── bases # References Base Repositories
│ └── ...
├── prod
│ └── ...
├── staging
│ └── ...
└── sandbox
└── ...The next step is applying the resources to your kubernetes cluster. Kustomize has a great CLI tool for building out the manifests. You run:
kustomize build sandbox/todoapp | kubectl apply f -Then applying the same app to production, but with, for example, a different endpoint:
kustomize build production/todoapp | kubectl apply f - What we liked about Kustomize was that it combined security best practices with simple extensibility. We recommend it to anyone building out a multi-tenant cluster.
A look to the future.
Cloud infrastructure is an exciting space to get immersed in. Though the number of options can seem overwhelming, it proves itself time and time again.
The community surrounding the space is fantastic. People and organisations are building amazing tooling and open-sourcing it for the benefit of the community. On the topic of community, we recommend checking out the following slack channels:
Building out your cloud infrastructure is a long, iterative process. No one can tell you the right way in a single blog post.
Start building out your tooling. Make the mistakes we did and improve.
In a future blog post, we will be talking about how we moved to a comprehensive CI/CD platform and which tools we used.
For now though, we’d like to thank you for reading till the end! We hope you enjoyed learning about an SME’s journey into cloud and hope you’ll continue to follow us as we improve.
Tom Burton is a DevOps engineer at Alteos.
Originally a Chemical Engineering graduate, Tom decided to switch to a coding career after attending a Blockchain party in a penthouse suite of the Sheraton New York. Upon completing a 9 week bootcamp, Tom explored many avenues of software development before choosing a career as a DevOps. He currently lives in Berlin and is working on creating a fully automated CI/CD pipeline at Alteos.
