Using Kubernetes + HaProxy to Host Scalable CTF challenges

A step by step guide on how to deploy and load test scalable containers on a k8 cluster!

https://kubernetes.io/

There is only one thing that CTF participants hate more than a boring CTF: a CTF with challenges that keep going down :)

Deploying CTF challenges is different from any normal server deployment or DevOps job because you’re intentionally deploying services that are vulnerable to break, and you got to be ready with fallback measures to minimize downtime when that does happen.

That’s what we had in mind while we picked using Kubernetes to deploy challenges for csictf 2020. We wanted to ensure that:

1. The challenges are deployed in a scalable manner. It should be trivial for us to scale up/scale down resources for a challenge in response to dynamic traffic.
2. The load has to be equally balanced between multiple instances of a challenge.
3. If a challenge does go down, we should have a strategy to quickly bring it up again, back to its initial state.

In this article, we’ll go over how you can set up a Kubernetes cluster to deploy challenges in such a manner, as to satisfy exactly these goals.

A quick refresher on Kubernetes terminology

(Skip ahead to the next section if you already know about k8 deployments, pods, and services)

A Kubernetes cluster consists of nodes and deployments.

Nodes are physical machines running inside your cluster. For example, if you were using a cloud provider, each VM instance you make would be a single node on the cluster.

A deployment is an abstract term that refers to one or more running instances of a container you want to deploy on the cluster. In simple words, if you want to run a container (or multiple instances of a container) on your cluster, you create a deployment telling Kubernetes: “Hey, here’s my container’s image, I want you to pick nodes on the cluster and deploy my container onto these nodes”.

A deployment consists of pods. A pod is an actual running instance of your container. When you create a deployment, Kubernetes goes ahead and creates pods and assigns them to run on nodes on the cluster. The powerful thing about Kubernetes is that you can tell it how many pods you want a deployment to have, and it will take care of ensuring that those many pods are always running on your cluster, and are moreover, equally distributed between nodes. In Kubernetes, this is referred to as ensuring that the cluster always has “minimum availability”.

Once you have pods running on the cluster, you need a way to expose these containers running on the cluster to the outside world. A service does just that. There are three kinds of services in k8: Node Ports, Load Balancers, and Cluster IPs. We will be using just Node Ports in this article, but in brief, a node port tells Kubernetes, “Hey, can you expose a port on all nodes the cluster and link that port to pods running under a deployment? And also make sure that load is equally distributed between all the pods :)” This can be hard to wrap your head around, but I recommend this great article to understand k8 services. Here’s a nice diagram from that article about NodePorts:

Exposing Port 30000 as a node port to pods on a cluster. (Source: https://medium.com/google-cloud/kubernetes-nodeport-vs-loadbalancer-vs-ingress-when-should-i-use-what-922f010849e0)

Setting up a K8 cluster on Google Cloud

The first thing you want to do is to provision a cluster on your cloud provider. We’ll be going or how to do this on GCP, but this should be possible on any provider in general.

Instructions also available on the official google cloud docs

You want to start by installing the Google Cloud SDK:

Make sure you have a google cloud project setup (you can create one here) and enable Google Kubernetes Engine for that project

Now run gcloud init and authenticate the CLI with your google cloud account. Make sure to set the default project as the project you created above, and the zone as default GCP zone.

Now, let’s create the cluster!

First, decide how many nodes you want on your cluster, what is the size of each of these nodes (the machine type), and which region are the nodes going to run on. If you want help deciding, refer to this article in our series, with LOADS of statistics from our CTF.

For the size of each node, you can run the following command to list all possible machine types (refer to google cloud’s docs for details about these types)

gcloud compute machine-types list

To list the possible regions, you can run the following command

gcloud compute zones list

Once you have these planned, run the following command to create the cluster:

Note the tags option, these will assign a tag to each VM instance on the node. This is very important, as we’ll be creating firewall rules later to expose ports on these nodes, and the tags will help us to target just instances on the cluster.

Now you should have a GKE cluster setup, let’s deploy a sample challenge to the cluster.

Using kubectl to deploy challenges on the cluster

First things first, you have to ensure that all your challenges are containerized. You can refer to this article in our series to setup Dockerfiles for CTF challenges.

We’re going to assume you have a docker image with the name challenge-image setup locally in the next few steps.

First, you need to push the image to a registry. If you’re using GCP, you can use the Google Container Registry, or even Github provides a free private registry.

For GCR, you can push the image as such:

gcloud auth configure-docker

docker tag challenge-image gcr.io/project-id/challenge-image

docker push gcr.io/project-id/challenge-image

Once you have the image pushed to a registry, we need to create a k8 deployment to deploy this image on our cluster.

For this, we need to create a deployment.yml file describing our deployment, and a k8 service to expose that deployment using a NodePort:

Refer to the comments in the file for an explanation of each section, and don’t forget to change the challenge-name and challenge-category , the ports exposed, the image URL, the no of replicas, etc. to match your use case.

Once you have the ymlfile setup, you can now deploy it to the cluster with a simple command:

kubectl apply -f deployment.yml

Verify that the deployment and service are running by using (note that you can use -l, in general, to filter by any label you created!)

kubectl get deployments,services -l challenge=challenge-name

An example challenge deployment from csictf, running 2 replicas

And that’s all there is to it, the challenge is now running on the cluster. Pick any node from your cluster, get its external IP, and try navigating to IP:NodePort (Where NodePort is the nodePort value you set in the yml file). You should see your challenge running on that port!

Note that no matter which node’s IP you use, k8 will take care of routing the request to a node that is running the pod!

Note: You will need to expose firewall ports on your cloud provider if it by default blocks incoming connections on all ports. In the case of gcp if you followed the instructions in the previous section, we can use gcloud to apply a firewall rule to allow port 30001(for example) on all nodes with the tag challenges :

Deploying more challenges

Just follow the same procedure above for each challenge, create a deployment.yml file, and use kubectl apply to deploy the challenge. Make sure to update the labels and name for each deployment/service, otherwise, you might overwrite on on top of an existing challenge!

List of deployments from our CTF, csictf 2020

If you set labels for challenge categories as we did in the same yml file above, you can also filter by a category, and perform operations on just a subset of the deployments, which is very handy during the CTF!

Example 1: Viewing port numbers for all “pwn” challenges

Example 2: Restart all containers running “web” challenges

Applying updates/changes to a deployment

The beautiful part about the apply command in the previous section is that later if you want to make changes to the same deployment (for example, update the tag of the image to push changes to the challenge, or changing the no of replicas), just modify the yml file, and as long as you have the same labels to match the deployment, k8 will apply the changes to the same deployment when you run the command again.

Sometimes, making changes like updating the port exposed might cause a conflict that k8 can’t handle, and it will throw an exception. In that case, first, delete the deployment with

kubectl delete -f deployment.yml

And then apply the yml file again

kubectl apply -f deployment.yml

Note:

You may have noticed that this deployment process gets a bit hard to manage as you have more and more challenges, as you have one yml file per challenge. We built a CLI tool just to automate this process of creating a deployment and a service. Refer to this article on ctfup and CI/CD in our series to know more about how to use it, or how you can build a similar tool for your use case!

Load Balancing between Nodes and Rate limiting

You may have noticed that currently, we accessed the deployment by accessing a single node’s IP and let Kubernetes then route the connection to the right node. But this is susceptible to an attacker overwhelming a single node with a lot of packets. K8 would still route the connections in a round-robin fashion to pods on the cluster, but the real issue is that it’s possible for an attacker to still overwhelm a single node with a lot of network requests.

There are several solutions to fix this issue:

1. Instead of NodePort you can use a LoadBalancer k8 service. This means that your cloud provider will handle the load balancing between nodes for you. The main issue with this approach is though, creating one load balancer rule per challenge can get costly. (For our 4 day CTF, we estimated 50$ would be spent if we went ahead with this, just on load balancer rules)
2. You can handle load balancing on the DNS level, by creating multiple A records against the same domain name (round-robin DNS). But a more persistent attacker could still just access one node by obtaining its IP address!
3. You can roll out your own VM instance running a reverse proxy like Nginx, or HaProxy to balance the load between nodes. This is the option we went with in our CTF, as it also sets up rate limiting for each challenge :)

This is completely optional, but if you want to set up such a load balancing solution too, you can refer to the next section, but most CTFs probably can get away without doing this too.

Setting up a HaProxy Load Balancer in front of your cluster

HAProxy

Start by provisioning another VM which will act as a reverse proxy to your challenges cluster. You can generally use a really small sized machine for this (1vCPU or lesser, 500MB-1GB RAM), as all this machine is doing is routing requests :).

Install HaProxy on the machine:

sudo apt update

sudo apt install haproxy

Edit /etc/default/haproxy and append ENABLED=1 to the file to enable HaProxy

nano /etc/default/haproxy

Now, let’s set up a HaProxy config file at /etc/haproxy/haproxy.cfg :

I’ve left comments in the file explaining what each section does, don’t forget to modify it according to your needs.

Once the config file is set up, just run

sudo systemctl restart haproxy

And HaProxy should now be running and load balancing+rate limiting connections to your challenges! (Make sure you have opened the required firewall ports on the machine running HaProxy too)

Bonus: Load testing your cluster

The best way to load test your cluster, it to attack it using an army of pods from another Kubernetes cluster :)

This is mostly out of scope from this article, but I recommend reading this great tutorial in google cloud’s official docs. We used the same process before our CTF, using locust (a load testing framework) running on top of a GKE cluster to raid some challenges with requests, so we can get an idea of how many replicas for each challenge would be “enough” during the CTF.

You can refer to this GitHub issue on our repo, where we posted some results from our load testing.

You reached The End

In this article, we went over how you can setup a k8 cluster to deploy CTF challenges on, and also setting up HaProxy to load balance connections to nodes on the cluster. If you’re interested in more aspects of hosting a CTF, like setting up CI/CD to deploy challenges, or on statistics/budget planning from a real CTF, do refer to other articles in our series below!

csictf