Validate if kubernetes deployment have LivenessProbe and ReadinessProbe enabled

Kubernetes is a container orchestration system that has become increasingly popular among companies looking for scalable and efficient solutions for managing their cloud applications. With the growing adoption of Kubernetes comes the need to ensure that the applications running on the cluster are always available and functioning correctly.

One way to ensure that the applications are always available is through the use of probes. Probes are mechanisms that allow Kubernetes to check if an application is functioning correctly. There are two types of probes in Kubernetes: LivenessProbe and ReadinessProbe.

The LivenessProbe is responsible for checking if the application is functioning correctly. If the probe detects that the application is not functioning correctly, Kubernetes will restart the pod that is running the application. This ensures that the application is restarted in case of failure, keeping it always available to users.

The ReadinessProbe, on the other hand, is responsible for checking if the application is ready to receive traffic. If the probe detects that the application is not ready to receive traffic, Kubernetes will remove the pod from the list of endpoints of the corresponding service. This ensures that traffic is not directed to an application that is not ready to receive it, thus avoiding failures in service delivery to users.

However, ensuring that probes are enabled in a deployment can be a challenging task, especially when working with multiple deployments in a Kubernetes cluster. This is where the admission webhook comes in.

The admission webhook is a mechanism that allows cluster administrators to define policies to control Kubernetes behavior. It is possible to create an admission webhook that validates whether the probes are enabled in a deployment before it is deployed to the cluster. This ensures that all applications deployed to the cluster have probes enabled, thus ensuring that they are always available and functioning correctly.

Creating an admission webhook to validate probes in a deployment involves creating a service that exposes an endpoint that will be called by Kubernetes every time a deployment is created or updated. This endpoint will receive information about the deployment being created or updated and then check if the probes are enabled. If the probes are not enabled, the endpoint will return an error and the deployment will not be deployed to the cluster.

In summary, using an admission webhook to validate whether probes are enabled in a deployment is an efficient way to ensure the availability and proper functioning of applications in a Kubernetes cluster. Additionally, this approach helps simplify cluster management as it allows policies to be defined centrally and automatically applied to all deployments.

Below i will show step by step how to configure it on your cluster using a python application.

1. Create a Docker image of your REST API server and push it to a container registry, such as Docker Hub.

app.py

from flask import Flask, make_response, jsonify, request
import os
import json
import logging
cert = os.environ.get('TLS_CRT', '/certs/tls.crt')
key = os.environ.get('TLS_KEY', '/certs/tls.key')

app = Flask(__name__)
FORMAT = 'Information - %(asctime)s %(message)s'
logging.basicConfig(level=logging.INFO, format=FORMAT)

@app.route('/health-check')
def health_check():
    response = make_response('OK', 200)
    return response

@app.route("/validate", methods=["POST"])
def validate():
    request_json = request.get_json()
    # requestjson = json.loads(str(request_json))
    deployment = request_json["request"]["object"]["spec"]["template"]["spec"]["containers"]
    # return deployment
    logging.info("Request JSON: %s", deployment)


    for container in deployment:
        if "livenessProbe" in container and "readinessProbe" in container:
            response = {
                "apiVersion": "admission.k8s.io/v1",
                "kind": "AdmissionReview",
                "response": {
                    "uid": request_json["request"]["uid"],
                    "allowed": True,
                }
            }
            logging.info("Response: %s", response)
            return jsonify(response)
        else:
            response = {
                "apiVersion": "admission.k8s.io/v1",
                "kind": "AdmissionReview",
                "response": {
                    "uid": request_json["request"]["uid"],
                    "allowed": False,
                    "status": {"message": "ReadinessProbe and LivenessProbe are required "}
                }
            }
            logging.info("Response: %s", response)
            return jsonify(response)

if __name__ == "__main__":
   app.run(host='0.0.0.0', port=8443, ssl_context=(cert, key))
    #app.run(host='0.0.0.0', port=8443)

Requirements.txt

Flask

dockerfile

FROM python:3.11.1-bullseye
RUN apt update
RUN apt upgrade -y

RUN mkdir -p admission-webhook
COPY app.py /admission-webhook
COPY requirements.txt /admission-webhook

RUN useradd -ms /bin/bash app_user
WORKDIR /admission-webhook
RUN python3 -m venv /admission-webhook/venv
RUN /admission-webhook/venv/bin/pip3 install -r /admission-webhook/requirements.txt
USER app_user
CMD /admission-webhook/venv/bin/python3 app.py

2. Generate a CA, sign a certificate, and generate a caBundle.

export CN=<yourservicename>.svc.cluster.local
export O=teste
export CITY=SP
export STATE=SP
export CERTIFICATE_NAME=validate-hc
export SSL_ALTNAME="subjectAltName=DNS:<yourservicename>.svc.cluster.local,DNS:<yourservicename>.svc,DNS:<yourservicename>.admission-webhook.svc"

# Gera a PK da rootCA
openssl genrsa -out rootCA.key 4096
# Gera o certificado auto assinado 
openssl req -x509 -new -nodes -key rootCA.key -days 1024 -subj "/C=$CITY/ST=$STATE/O=$O, Inc./CN=$CN" -out rootCA.crt
# Gera a pk
openssl genrsa -out $CERTIFICATE_NAME.key 2048

#Gera um certificado Request
openssl req -new -sha256 -key $CERTIFICATE_NAME.key -subj "/C=$CITY/ST=$STATE/O=$O, Inc./CN=$CN" -out $CERTIFICATE_NAME.csr

#Assina o certificado com a chave da CA
# openssl x509 -req -in $CERTIFICATE_NAME.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out $CERTIFICATE_NAME.crt -days 500
openssl x509 -req -extfile <(printf "$SSL_ALTNAME") -days 365 -in $CERTIFICATE_NAME.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out $CERTIFICATE_NAME.crt


#concatena o certificado

cat rootCA.crt > caBundle.pem
cat $CERTIFICATE_NAME.crt >> caBundle.pem

#openssl x509 -in caBundle.pem -text -noout
openssl x509 -req -extfile <(printf "$SSL_ALTNAME") -days 365 -in $CERTIFICATE_NAME.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out $CERTIFICATE_NAME.crt

3. Create a new Namespacein your cluster to deploy admission-webhook.

kubectl create ns admission-webhook

4. Create a Secretin your cluster to store the certificate and private key created before and deploy it inside namespace created before.

apiVersion: v1
kind: Secret
metadata:
  name: webhook-certs
type: Opaque
data:
  tls.crt: <Base64 Certificate>
  tls.key: <Base64 PrivateKey>

5. Create a deployment in your cluster that runs the Docker image. You can use a Deployment object in Kubernetes to manage the replicas of your REST API server deploy it inside namespace created before.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: admission-webhook
spec:
  replicas: 1
  selector:
    matchLabels:
      app: validate-probes-webhook
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: validate-probes-webhook
    spec:
      containers:
      - name: admission-webhook
        image: <register/repo:tag>
        ports:
        - containerPort: 8443
        readinessProbe:
          httpGet:
            scheme: HTTPS
            path: /health-check
            port: 8443
          initialDelaySeconds: 5
          timeoutSeconds: 2
        livenessProbe:
          httpGet:
            scheme: HTTPS
            path: /health-check
            port: 8443
          initialDelaySeconds: 15
          timeoutSeconds: 5
        securityContext:
          allowPrivilegeEscalation: false
          runAsNonRoot: true
          runAsUser: 2000
        resources:
          limits:
            cpu: 200m
            memory: 512Mi
          requests:
            cpu: 50m
            memory: 128Mi
        volumeMounts:
          - name: webhook-certs
            mountPath: /certs
            readOnly: true
      volumes:
        - name: webhook-certs
          secret:
            secretName: webhook-certs

6. Expose the REST API server to the network by creating a Service object. This will create a stable IP address and DNS name for your REST API server, allowing the Kubernetes API server to reach it.

apiVersion: v1
kind: Service
metadata:
  name: validate-hc
  namespace: <your-namespace>
spec:
  selector:
    app: validate-probes-webhook
  ports:
  - name: http
    port: 443
    targetPort: 8443
  type: ClusterIP

7. Register the admission webhook with the Kubernetes API server by creating a ValidatingWebhookConfiguration object. This object tells the API server about your webhook, including the endpoint where the API server should send requests, the types of resources it should be applied to, and the criteria for triggering the webhook.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: "<NAMEOFYOURSERVICE>.svc.cluster.local"
webhooks:
  - name: "<NAMEOFYOURSERVICE>.svc.cluster.local"
    admissionReviewVersions: [ "v1" ]
    sideEffects: None
    namespaceSelector:
      matchExpressions:
        - key: kubernetes.io/metadata.name
          values: ["namespace1", "namespace2"] #YOU CAN USE THIS AS WHITELIST
          operator: NotIn
    rules:
      - apiGroups: ["*"]
        apiVersions: ["v1"]
        operations: ["CREATE","UPDATE"]
        resources: ["deployments"]
        scope: "*"
    clientConfig:
      service:
        namespace: <NAMESPACE-OF-YOUR-SERVICE>
        name: <NAME-OF-YOUR-SERVICE>
        path: /validate
        port: 443
      caBundle: <YOUR-CABUNDLE-HERE>

After deploy everything we can use test it with the deployment below.

kubectl create ns api-test

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api-test
  template:
    metadata:
      labels:
        app: api-test
        admission-webhook: enabled
    spec:
      containers:
      - name: api-test
        image: herrmann89/api-test:latest
        ports:
        - containerPort: 8080

After apply the yaml above we should see this error in deployment.