A step by step guide on how to deploy and load test scalable containers on a k8 cluster!
There is only one thing that CTF participants hate more than a boring CTF: a CTF with challenges that keep going down :)
Deploying CTF challenges is different from any normal server deployment or DevOps job because you’re intentionally deploying services that are vulnerable to break, and you got to be ready with fallback measures to minimize downtime when that does happen.
That’s what we had in mind while we picked using Kubernetes to deploy challenges for csictf 2020. We wanted to ensure that:
- The challenges are deployed in a scalable manner. It should be trivial for us to scale up/scale down resources for a challenge in response to dynamic traffic.
- The load has to be equally balanced between multiple instances of a challenge.
- If a challenge does go down, we should have a strategy to quickly bring it up again, back to its initial state.
In this article, we’ll go over how you can set up a Kubernetes cluster to deploy challenges in such a manner, as to satisfy exactly these goals.
A quick refresher on Kubernetes terminology
(Skip ahead to the next section if you already know about k8 deployments, pods, and services)
A Kubernetes cluster consists of nodes and deployments.
Nodes are physical machines running inside your cluster. For example, if you were using a cloud provider, each VM instance you make would be a single node on the cluster.
A deployment is an abstract term that refers to one or more running instances of a container you want to deploy on the cluster. In simple words, if you want to run a container (or multiple instances of a container) on your cluster, you create a deployment telling Kubernetes: “Hey, here’s my container’s image, I want you to pick nodes on the cluster and deploy my container onto these nodes”.
A deployment consists of pods. A pod is an actual running instance of your container. When you create a deployment, Kubernetes goes ahead and creates pods and assigns them to run on nodes on the cluster. The powerful thing about Kubernetes is that you can tell it how many pods you want a deployment to have, and it will take care of ensuring that those many pods are always running on your cluster, and are moreover, equally distributed between nodes. In Kubernetes, this is referred to as ensuring that the cluster always has “minimum availability”.
Once you have pods running on the cluster, you need a way to expose these containers running on the cluster to the outside world. A service does just that. There are three kinds of services in k8: Node Ports, Load Balancers, and Cluster IPs. We will be using just Node Ports in this article, but in brief, a node port tells Kubernetes, “Hey, can you expose a port on all nodes the cluster and link that port to pods running under a deployment? And also make sure that load is equally distributed between all the pods :)” This can be hard to wrap your head around, but I recommend this great article to understand k8 services. Here’s a nice diagram from that article about NodePorts:
Setting up a K8 cluster on Google Cloud
The first thing you want to do is to provision a cluster on your cloud provider. We’ll be going or how to do this on GCP, but this should be possible on any provider in general.
You want to start by installing the Google Cloud SDK:
gcloud init and authenticate the CLI with your google cloud account. Make sure to set the default project as the project you created above, and the zone as default GCP zone.
Now, let’s create the cluster!
First, decide how many nodes you want on your cluster, what is the size of each of these nodes (the machine type), and which region are the nodes going to run on. If you want help deciding, refer to this article in our series, with LOADS of statistics from our CTF.
For the size of each node, you can run the following command to list all possible machine types (refer to google cloud’s docs for details about these types)
gcloud compute machine-types list
To list the possible regions, you can run the following command
gcloud compute zones list
Once you have these planned, run the following command to create the cluster:
Note the tags option, these will assign a tag to each VM instance on the node. This is very important, as we’ll be creating firewall rules later to expose ports on these nodes, and the tags will help us to target just instances on the cluster.
Now you should have a GKE cluster setup, let’s deploy a sample challenge to the cluster.
Using kubectl to deploy challenges on the cluster
First things first, you have to ensure that all your challenges are containerized. You can refer to this article in our series to setup Dockerfiles for CTF challenges.
We’re going to assume you have a docker image with the name
challenge-image setup locally in the next few steps.
First, you need to push the image to a registry. If you’re using GCP, you can use the Google Container Registry, or even Github provides a free private registry.
For GCR, you can push the image as such:
gcloud auth configure-docker
docker tag challenge-image gcr.io/project-id/challenge-image
docker push gcr.io/project-id/challenge-image
Once you have the image pushed to a registry, we need to create a k8 deployment to deploy this image on our cluster.
For this, we need to create a
deployment.yml file describing our deployment, and a k8 service to expose that deployment using a NodePort:
Refer to the comments in the file for an explanation of each section, and don’t forget to change the
challenge-category , the ports exposed, the image URL, the no of replicas, etc. to match your use case.
Once you have the
ymlfile setup, you can now deploy it to the cluster with a simple command:
kubectl apply -f deployment.yml
Verify that the deployment and service are running by using (note that you can use -l, in general, to filter by any label you created!)
kubectl get deployments,services -l challenge=challenge-name
And that’s all there is to it, the challenge is now running on the cluster. Pick any node from your cluster, get its external IP, and try navigating to
IP:NodePort (Where NodePort is the
nodePort value you set in the
yml file). You should see your challenge running on that port!
Note that no matter which node’s IP you use, k8 will take care of routing the request to a node that is running the pod!
Note: You will need to expose firewall ports on your cloud provider if it by default blocks incoming connections on all ports. In the case of
gcp if you followed the instructions in the previous section, we can use
gcloud to apply a firewall rule to allow port
30001(for example) on all nodes with the tag
Deploying more challenges
Just follow the same procedure above for each challenge, create a
deployment.yml file, and use
kubectl apply to deploy the challenge. Make sure to update the labels and name for each deployment/service, otherwise, you might overwrite on on top of an existing challenge!
If you set labels for challenge categories as we did in the same
yml file above, you can also filter by a category, and perform operations on just a subset of the deployments, which is very handy during the CTF!
Applying updates/changes to a deployment
The beautiful part about the
apply command in the previous section is that later if you want to make changes to the same deployment (for example, update the tag of the image to push changes to the challenge, or changing the no of replicas), just modify the
yml file, and as long as you have the same labels to match the deployment, k8 will apply the changes to the same deployment when you run the command again.
Sometimes, making changes like updating the port exposed might cause a conflict that k8 can’t handle, and it will throw an exception. In that case, first, delete the deployment with
kubectl delete -f deployment.yml
And then apply the
yml file again
kubectl apply -f deployment.yml
You may have noticed that this deployment process gets a bit hard to manage as you have more and more challenges, as you have one
yml file per challenge. We built a CLI tool just to automate this process of creating a deployment and a service. Refer to this article on ctfup and CI/CD in our series to know more about how to use it, or how you can build a similar tool for your use case!
Load Balancing between Nodes and Rate limiting
You may have noticed that currently, we accessed the deployment by accessing a single node’s IP and let Kubernetes then route the connection to the right node. But this is susceptible to an attacker overwhelming a single node with a lot of packets. K8 would still route the connections in a round-robin fashion to pods on the cluster, but the real issue is that it’s possible for an attacker to still overwhelm a single node with a lot of network requests.
There are several solutions to fix this issue:
- Instead of
NodePortyou can use a
LoadBalancerk8 service. This means that your cloud provider will handle the load balancing between nodes for you. The main issue with this approach is though, creating one load balancer rule per challenge can get costly. (For our 4 day CTF, we estimated 50$ would be spent if we went ahead with this, just on load balancer rules)
- You can handle load balancing on the DNS level, by creating multiple A records against the same domain name (round-robin DNS). But a more persistent attacker could still just access one node by obtaining its IP address!
- You can roll out your own VM instance running a reverse proxy like Nginx, or HaProxy to balance the load between nodes. This is the option we went with in our CTF, as it also sets up rate limiting for each challenge :)
This is completely optional, but if you want to set up such a load balancing solution too, you can refer to the next section, but most CTFs probably can get away without doing this too.
Setting up a HaProxy Load Balancer in front of your cluster
Quick News Recent News Description Main features Supported Platforms Performance Reliability Security Download…
Start by provisioning another VM which will act as a reverse proxy to your challenges cluster. You can generally use a really small sized machine for this (1vCPU or lesser, 500MB-1GB RAM), as all this machine is doing is routing requests :).
Install HaProxy on the machine:
sudo apt update
sudo apt install haproxy
/etc/default/haproxy and append
ENABLED=1 to the file to enable HaProxy
Now, let’s set up a HaProxy config file at
I’ve left comments in the file explaining what each section does, don’t forget to modify it according to your needs.
Once the config file is set up, just run
sudo systemctl restart haproxy
And HaProxy should now be running and load balancing+rate limiting connections to your challenges! (Make sure you have opened the required firewall ports on the machine running HaProxy too)
Bonus: Load testing your cluster
The best way to load test your cluster, it to attack it using an army of pods from another Kubernetes cluster :)
This is mostly out of scope from this article, but I recommend reading this great tutorial in google cloud’s official docs. We used the same process before our CTF, using locust (a load testing framework) running on top of a GKE cluster to raid some challenges with requests, so we can get an idea of how many replicas for each challenge would be “enough” during the CTF.
You reached The End
In this article, we went over how you can setup a k8 cluster to deploy CTF challenges on, and also setting up HaProxy to load balance connections to nodes on the cluster. If you’re interested in more aspects of hosting a CTF, like setting up CI/CD to deploy challenges, or on statistics/budget planning from a real CTF, do refer to other articles in our series below!