Using Kwok to Simulate Large Kubernetes Clusters

This post explores how to get started and setup project kwok to simulate large clusters for testing purposes.

When creating tools for the cloud-native ecosystem, it is often necessary to create large clusters ensure that your tool scales properly when the number of nodes and other resources grow beyond typical values. This was the case recently when I needed to tweak the UI for project ktop.

Learn more about Kwok

Kwok (or Kubernetes WithOut Kubelet) is a cluster simulator that allows you to create large clusters for testing purposes. It uses the standard Kubernetes API to deploy large number of resources in seconds without actually creating real nodes and pods.

The main component of kwok is a Kubernetes controller that simulates the cluster resources including nodes and pods. This can be done on an existing Kubernetes cluster (i.e. Kind) or you can use the provided kwokctl tool to create your own kind-like cluster.

Getting started

The project is fairly new, so this post will focus on using Homebrew for installation. If you are not using Homebrew, then you will need to go through a few steps (see Kwok installation instructions).

brew install kwok

This will install two binaries on your local machine:

• kwokctl — a CLI management too that allows you to standup clusters using Docker (similar to a Kind cluster).
• kwok — the resource controller that is responsible for emulating the cluster and its resources.

Creating a cluster

To simulate a fake cluster locally, ensure your Docker daemon is up and running. Then use the following command to create your cluster:

kwokctl create cluster --name kwok-cluster

This will download and run the necessary Docker images to setup your new cluster. You can verify that with the following:

$> docker ps
CONTAINER ID   IMAGE                                             COMMAND                  CREATED          STATUS          PORTS                                NAMES
aba80a50803A   registry.k8s.io/kwok/kwok:v0.1.1                  "kwok --manage-all-n…"   11 minutes ago   Up 11 minutes                                        kwok-kwok-cluster-kwok-controller
7df468aaecf7   registry.k8s.io/kube-controller-manager:v1.26.0   "kube-controller-man…"   11 minutes ago   Up 11 minutes                                        kwok-kwok-cluster-kube-controller-manager
ac2d44e4f259   registry.k8s.io/kube-scheduler:v1.26.0            "kube-scheduler --ku…"   11 minutes ago   Up 11 minutes                                        kwok-kwok-cluster-kube-scheduler
2bbe46094c41   registry.k8s.io/kube-apiserver:v1.26.0            "kube-apiserver --ad…"   11 minutes ago   Up 11 minutes   0.0.0.0:32766->6443/tcp              kwok-kwok-cluster-kube-apiserver
2eb999726db3   registry.k8s.io/etcd:3.5.6-0                      "etcd --name=node0 -…"   11 minutes ago   Up 11 minutes   2379-2380/tcp, 4001/tcp, 7001/tcp

Validate that the cluster is created:

$> kwokctl get clusters
kwok-cluster

Start the resource controller

Next, you will need to start the kwok resource controller. You can do this by running the kwok binary (as demo’d here) in a separate terminal. You can also run the controller in-cluster (see kwok doc for detail).

$> kwok \
  --kubeconfig=~/.kube/config \
  --manage-all-nodes=true \
  --cidr=10.0.0.1/24 \
  --node-ip=10.0.0.1

The previous command starts the kwok resource controller and specifies that controller will manage all nodes as Kwok nodes. Since the cluster is not shared with another cluster, that makes sense. Otherwise, you would need to add additional parameters to filter out Kwok nodes and resources.

Adding nodes

Next, create a file with your node definitions as shown below. You can add as many nodes, to your YAML file, that you would like to simulate.

apiVersion: v1
items:
- apiVersion: v1
  kind: Node
  metadata:
    annotations:
      node.alpha.kubernetes.io/ttl: "0"
      kwok.x-k8s.io/node: fake
    labels:
      beta.kubernetes.io/arch: arm64
      beta.kubernetes.io/os: linux
      kubernetes.io/arch: arm64
      kubernetes.io/hostname: kwok-node-0
      kubernetes.io/os: linux
      minikube.k8s.io/name: kwok-node-0
      minikube.k8s.io/primary: "true"
      node-role.kubernetes.io/control-plane: ""
      node-role.kubernetes.io/master: ""
      type: kwok
    name: kwok-node-0
  spec:
    taints: # Avoid scheduling 
      - effect: NoSchedule
      key: kwok.x-k8s.io/node
      value: fake
  status:
    allocatable:
      cpu: "4"
      memory: 512Gi
      pods: "110"
    capacity:
      cpu: "4"
      memory: 512Gi
      pods: "110"
    nodeInfo:
      architecture: arm64
      kubeProxyVersion: fake
      kubeletVersion: fake
      operatingSystem: linux
---
### Add more nodes

Next, we need to apply the nodes to the cluster. Before you do that, ensure you are using the right cluster context:

$> kubectl config use-context kwok-kwok-cluster

Then, use your kubectl tool to apply the nodes:

kubectl apply -f nodes.yaml

Once the YAML is applied successfully, you can see your nodes in the cluster:

$> kubectl get nodes
NAME          STATUS   ROLES   AGE   VERSION
kwok-node-0   Ready    agent   32s   fake
kwok-node-1   Ready    agent   32s   fake
kwok-node-2   Ready    agent   32s   fake
kwok-node-3   Ready    agent   32s   fake
kwok-node-4   Ready    agent   32s   fake
kwok-node-5   Ready    agent   32s   fake
kwok-node-6   Ready    agent   32s   fake

Deploy resources

Now that you have a kwok cluster running with several fake nodes, you can deploy resources to it, as if it is a real cluster.

Define a deployment with a large number of replicas. Note the markup on the spec and affinity definitions to ensure that the kwok controller properly simulate the deployments.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: fake-web-1
  namespace: default
spec:
  replicas: 100
  selector:
    matchLabels:
      app: fake-web-1
  template:
    metadata:
      labels:
        app: fake-web-1
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: type
                    operator: In
                    values:
                      - kwok
      # A taints was added to an automatically created Node.
      # You can remove taints of Node or add this tolerations.
      tolerations:
        - key: "kwok.x-k8s.io/node"
          operator: "Exists"
          effect: "NoSchedule"
      containers:
        - name: fake-container
          image: fake-image

---
## Add as many resources as needed

As before, you can add as many fake resources as you desire. Next, apply the YAML file as shown below:

$> kubectl apply -f deployments.yaml

Within seconds, you will see your fake resources show up in the cluster:

$> kubectl get deployments
NAME             READY     UP-TO-DATE   AVAILABLE   AGE
fake-backend-1   25/25     25           25          42s
fake-backend-2   10/10     10           10          42s
fake-web-1       100/100   100          100         42s
fake-web-2       50/50     50           50          42s

Kwok with ktop

Before kwok, I would need to stand up real clusters with large numbers of resources to test ktop 🎉

ktop displaying fake kwok simulated cluster resources

Conclusion

Kwok is a great addition to the cloud-native ecosystem, especially if you are building tools using the Kubernetes API. I am sure as more contributors adopt the tool, it will get better. Looking forward to use it in other projects.