Kubernetes on Hetzner
Smartly.io automates every step of social advertising to unlock greater performance and creativity and is one of the brightest stars in the Finnish startup scene nowadays. The company has been growing very fast. Scaling the service and infrastructure to fulfill the needs of development teams so they can keep on shipping new awesome features (like video templating which is a huge feature from every point of view). Growing a company and the infrastructure at the same time is not an easy job, but Smartly.io have nailed it!
Scale the infrastructure with the company
When you scale the number of developers and teams, it requires scalability and agility from the underlying architecture, infrastructure, and development environments. Creating new service from scratch with the development environment, testing pipelines, and deploying it to production with all required resources (like servers, databases, domain names, TLS certificates), needs to be as easy, quick and automated as possible. The process should not require any work from the DevOps team or otherwise, it becomes the bottleneck for the scaling.
Polar Squad, for a young company, had a long history of running and managing Kubernetes in production. And we started a collaboration with Smartly.io to create architecture that can scale in the number of services and engineer needs.
Running Kubernetes in Hetzner
Hetzner provides dedicated servers and Smartly.io uses Hetzner heavily for hosting. 1000’s of dedicated servers provides the image and video rendering power what building next-generation advertisement platform needs. The cloud’s worse performance would require even larger infrastructure and costs would be sky high in traditional clouds.
This gave a nice challenge for our Kubernetes expertise, build a Kubernetes cluster to on-premise like environment!
If you want to run Kubernetes in the cloud, we highly suggest you consider first:
- hosted cluster — if you don’t have resources for maintaining the cluster and don’t need much customization
- Use automation tools like kops — if you’re ok with the fact that the architecture is not fully in your hands
- Use kubeadm — to get good default configurations
- Install the cluster with your automation tooling — to get the full control but remember this requires continuous maintenance
But these weren’t our options in Hetzner. In Hetzner we need to make the whole setup from scratch.
Installing Kubernetes is not hard, configuring, upgrading and scaling is!
There’s a lot of open source tooling to install the Kubernetes. Installing is not the hard part, in the end, it’s just a few configs, certificates, couple YAML-files, and systemd service configuration, that’s it!
We evaluated quite many open source tools but in the end, came to the conclusion that the installation with Ansible is not a big thing, the real challenge comes with the configuration. And the tools might give a good baseline, it requires anyway fine-tuning pretty quickly and we would need to get rid of the tool at that point. Smartly.io also has a highly talented DevOps team, and transparent setup is better than hiding details under some tool.
Take the time to configure Kubernetes properly
If there’s only one thing that you need to remember from this, remember that
by default, Kubernetes is not secure!
There’s a long list of best security practices what you should really go through to ensure it’s secure. Right from the start, we got hacked by some hacker and he/she was able to run bitcoin miner for a while. We were running our initial setup and had disabled quite many security layers just to take the smoke out from the cluster. It took a few hours for a hacker to detect our cluster and start the bitcoin miner in one of the servers.
Take the time and make sure your cluster is secured, don’t be the next company who got hacked like Tesla or Digital Ocean, just to name few.
No private network, secure cluster in public internet
In Hetzner by default, all the dedicated servers are just plain servers on the public internet, without any private networking. This means that all traffic in the cluster goes over the public internet and must be encrypted. By selecting an overlay network that supports encryption, we were able to encrypt the Pods traffic, and one by one go through enabling TLS encryption in each component. If you’re following the same practice, remember to also secure and encrypt traffic in the master nodes.
No cloud integration, no auto-scaling, no LBs
Obviously, you cannot spin up more servers automatically when you use dedicated servers. Getting a new server takes more like days than minutes, which is not common anymore in the nowadays clouded world, but it’s worth it when you need to do heavy computation and having virtual CPUs is not an option from money and performance ways.
With good monitoring and capacity planning, you can foresee the requirements and act based on it. Also, you can prioritize workload on Kubernetes so that in case of high load, not so critical services scales down.
I could see this kind of setup really good also for any data processing, machine learning, etc. use cases, where you want good performance, need a lot of resources, but the same time all goodies what Kubernetes brings.
Also if you have only used cloud hosted Kubernetes, you might not have realized a few features that the Kubernetes cloud integration does for you. The most notable one is type=LoadBalancer Service which creates external load balancer for your service and does the magic so traffic ends up to your Pods. I’ll share more details later on in another post, but you can get the same agility without cloud by using wildcard domains and doing load balancing and certificate termination in the Ingress controller.
Also if you run some stateful services in Kubernetes, like databases, storages, etc., getting new Volumes doesn’t come out of the box, like in the cloud. You need to implement the automation for creating new volumes to Kubernetes cluster, backups and the whole lifecycle of them.
Running only stateless services, which is quite common practice still, solve this problem by pushing the work somewhere else (e.g. use Database-As-a-Service if you don’t want to maintain the databases).
Not having cloud integration gives the benefit that we can easily move this to any environment and we’re really aware of the external requirements for the cluster.
Even if you use a cloud, you can force yourself to run it without the integration. I did it once in the past to ensure capability for moving to any cloud or on-premise environment. It wasn’t that hard, and I liked the transparency what it gave to me.
Conclusion
Running Kubernetes in Hetzner has been successful so far and this journey has deepened our expertise with Kubernetes. Self-hosting Kubernetes on Hetzner requires much more human resources compared to using hosted services, but it gives back your transparency, chance to learn, and save costs.
Polar Squad has a long history from Kubernetes in multiple different environments. But we’re are not just experts of specific cloud or technology,
we use technology to solve problems so that developers can do their job as easily, safely and quickly as possible. In the end, it’s the software what produces value to the end customer 😎
