Kubernetes Control Loops: The Secret Sauce Behind Your Microservice Bliss

TLDR; Ever dreamt of a system that manages your microservices like a maestro, constantly keeping them in perfect harmony? Kubernetes makes this a reality with control loops, the magic behind the scenes. Operators take it a step further, letting you customize Kubernetes for even more control over custom workload needs.

DALL-E | “create an image describing kubernetes Control Loop”

The 💪🏽 Power of the Control Loop ➿

Imagine your application as a complex symphony. Each microservice plays a vital role, but maintaining this harmony can be a nightmare. Kubernetes control loops come to the rescue! These are programmatic wizards that constantly monitor two things:

1. Desired State: This is defined by you using Kubernetes objects like Deployments. It specifies how many instances of your service you want running, their resource requirements, and desired configurations.
2. Actual State: This reflects the real-time state of your cluster. It includes the number of running pods (microservice instances), resource usage, etc .

The control loop acts like a conductor, continuously comparing the desired and actual states. If there’s a mismatch, it takes corrective actions. For example, if a Deployment specifies three replicas but only two are running, the control loop will spin up a new pod to match your desired state. This ensures your applications always run as intended, even amidst failures or scaling demands.

This image shows what the standard controllers come with kubernetes out of the box before we run to build our own … credit to RedHat

Control Loop in action

Let’s imaging a symphony orchestra preparing for a performance.

• The Score (Desired State): The sheet music represents the desired state of the symphony. It specifies the instruments needed, the notes each instrument plays, and the overall composition. This is analogous to a Kubernetes deployment defining the desired state of your application (container image, resource requests, replicas) which looks like the following:

## yaml generated by running the following kubectl command:
## kubectl create deployment --namespace default --image traefik/whoami whoami --replicas 2 --dry-run=client -oyaml
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: whoami
  name: whoami
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: whoami
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: whoami
    spec:
      containers:
      - image: traefik/whoami
        name: whoami
        resources: {}
status: {}

• The Conductor (Deployment Controller): The conductor oversees the entire orchestra, ensuring everyone plays according to the score. They constantly monitor the performance and make adjustments as needed. Similarly, the deployment controller watches for new deployments and creates replica sets to manage the desired application state.

kubectl get deployment

NAME     READY   UP-TO-DATE   AVAILABLE   AGE
whoami   2/2     2            2           8s

• Sections (Replica Sets): The violin section, brass section, etc., each represent a replica set. Each section is responsible for a specific group of instruments within the orchestra. Replica sets manage groups of pods (running instances) of your application (note the replica-set id in the ouput example 6f57d5d6b5 )

kubectl get rs
NAME                DESIRED   CURRENT   READY   AGE
whoami-6f57d5d6b5   2         2         2       10m

• Musicians (Pods): Individual musicians playing their instruments are like pods. Each pod is a runnable unit containing one or more containers that make up your application -> note the replica-set id is part of the pod name 6f57d5d6b5-xxxxx

kubectl get po
NAME                      READY   STATUS    RESTARTS   AGE
whoami-6f57d5d6b5-46p5l   1/1     Running   0          13m
whoami-6f57d5d6b5-ghpgn   1/1     Running   0          13m

• The Stage Manager (Replica Set Controller): The stage manager ensures the right number of musicians from each section are on stage and ready to play. They add or remove musicians as needed to follow the conductor’s instructions. The replica set controller functions similarly, managing the number of pods running for the application. It creates new pods if some are missing or terminates extras to match the desired state dictated by the deployment.

kubectl scale deployment --replicas 3 whoami
deployment.apps/whoami scaled

kubectl get po
NAME                      READY   STATUS    RESTARTS   AGE
whoami-6f57d5d6b5-46p5l   1/1     Running   0          15m
whoami-6f57d5d6b5-ghpgn   1/1     Running   0          15m
whoami-6f57d5d6b5-l5sx5   1/1     Running   0          14s

• Playing the Music (Kubelet): Each musician follows the conductor’s instructions and plays their part. The kubelet, running on each worker node (like a musician’s stand), receives instructions from the replica set controller to create or remove pods. It then manages the lifecycle of these pods on the node, ensuring they are running correctly.

Continuously Performing (the Loop):

The conductor (deployment controller) keeps referring to the score (deployment) to ensure the orchestra (application) adheres to the desired state. The stage manager (replica set controller) and the musicians (pods) on their stands (kubelet) work together to make sure the right number of musicians are playing their parts, following the conductor’s lead and achieving a harmonious performance (stable and healthy application running as intended).

This orchestration could be applied to custom resources and workflows which might be required in your symphoney …

Operators — Extending the Kubernetes Symphony

Control loops are powerful, but what if you need to manage something beyond vanilla Kubernetes resources? That’s where operators shine. Operators are custom controllers built using the same control loop principle. They watch for custom resources you define (think of them as new instruments for your symphony) and take actions to manage them.

DALL-E | Extending the Kubernetes Symphony

Let’s review a real-world example: imagine managing sensitive information for your applications securely. Secrets in kubernetes are just 64bit encoding (which isn’t encryption!), we need a secure way to store secrets and ensure that they are accessed / used in a secure way. Enters the external-secrets-operator. It watches for a custom resource defining your secrets and securely retrieves them at runtime (or just in time) from external sources like Vault or any cloud providers Secrets Manager (AWS/Google/Azure). It then injects these secrets into your pods using the control loop, ensuring your applications have access to sensitive information without it being stored directly in the container image.

🏗️ Custom Resource Definitions (CRDs) and Operator Magic ✨

Operators leverage Custom Resource Definitions (CRDs) to extend Kubernetes. CRDs are like blueprints for your custom resources, defining what information they hold and how they should be managed. The operator’s control loop then watches for these custom resources and performs actions based on their specifications. This allows you to tailor Kubernetes to your specific needs, adding functionalities like managing databases, monitoring tools, or even custom application lifecycles.

Visualizing the “control loop” looks like the following:

Credit | miro

🚀 When to Develop Your Own Operator ?

While operators offer immense flexibility, it’s important to understand when they’re truly necessary:

Consider developing an operator when:

• Managing complex resources: If you need to manage resources beyond vanilla Kubernetes objects, an operator provides a structured and scalable approach these resources could be databases, external secrets, or custom application deployments.
• Automating repetitive tasks: Operators excel at automating repetitive tasks specific to your environment. This could involve configuring resources, injecting secrets, or performing health checks.
• Promoting reusability: If you find yourself managing similar resources across multiple deployments, creating an operator allows you to package and reuse the logic, saving time and ensuring consistency.

Before diving into CRDs, remember:

• Start simple: Often, existing solutions or configuration management tools might suffice. Evaluate the complexity of your needs before embarking on operator development.
• Community first: Check if there’s already an operator available for your specific need. The Kubernetes community is vibrant, and you might find a pre-built solution that saves you development time.

I recommend you head over the https://operatorhub.io/ to view a long list of community driven operators which may help you take better advantage of the kubernetes control loop without maintaining your own operator, maintaining your own operator requires some coding skills (golang) which requires a profound understanding on how the operator SDK works, and how to integrate it into your cluster → something which might be discussed in future posts.

Conclusion — Microservices Nirvana Awaits 🌴

If you’re building microservices, Kubernetes with its control loop(s) is a game-changer. It automates mundane tasks, ensures application health, and scales seamlessly. Operators take it a step further, letting you manage even the most complex configurations with ease. So, ditch the manual toil and embrace the symphony of automation. Kubernetes awaits to make your microservice life a harmonious dream!

stay tuned for a follow-up post describing common Controllers which do 80% of what you might need to support your application (external-secrets-operator mentioned above was one of them).

Yours Sincerely,
Haggai Philip