Auto-Scaling Your Long Running Workloads on Kubernetes With KEDA v2
In one of my projects we use an event-based architecture to process new data as it arrives to make it available to our developers quickly. The data arrives in batches, and take anything from minutes to hours to process. Processing the data includes, for example, composing a video from individual images and saving it as a mp4 file, or performing a battery of tests created by our developers and analysts.
Our initial solution was based on Kubernetes Deployments that read events from GCP PubSub and invoked containerized code. We used Google’s Custom Metrics Adapter and the Horizontal Pod Autoscaler (HPA) to set the number of replicas to match the number of messages in a PubSub topic.
We soon identified a serious issue — busy pods got terminated during scale-down. As the number of messages in the queue decreases, the HPA automatically shuts down some of the pods. Since the HPA does not know which pods are currently doing actual work, it terminated pods in mid-work and forced them to restart We have seen cases where a 30 minutes job took hours because it kept restarting.
One way to address this issue is to trick Kubernetes into keeping your pods alive using a preStop hook, but this seemed too hacky. Another option is to use Kubernetes Jobs, which seems to fit the problem — a Job can handle a request to completion and exist. However, these jobs still need to be managed so they can be event-driven.
KEDA ScaledJob
Enters KEDA. KEDA is a Kubernetes-based Event Driven Autoscaler. With KEDA, you can drive the scaling of any container in Kubernetes based on the number of events needing to be processed.
KEDA does that in a clean, simple way — at its most basic configuration, it will make sure that the number of running Jobs matches the number of messages in the job queue. KEDA does not terminate running Jobs though, so you can rest assured that your long running workloads won’t get terminated. You can also set the maximum number of running jobs, control the job-to-messages ratio, polling interval and more.
Note that KEDA only controls the scaling of your jobs, but it is still the job’s responsibility to pull work from the queue. KEDA’s best practice is to have each Job read a single message from the queue, process it, remove it from the queue, and exit.
An example
In this example, we will use KEDA’s ScaledJob — a new kind of CRD introduced in the recently released version 2. For a job queue, we will use GCP PubSub (though KEDA supports many types of triggers). The initial set up steps are taken from the excellent KEDA Go PubSub example.
Prerequisites:
- A GCP account and a GCP account
- GCloud installed and configured to point to your project, as explained here
- A Kubernetes cluster, e.g. — Minikube or GKE
- KEDA 2.0 installed on your Kubernetes cluster, as explained here
First, we set up a GCP service account. KEDA will use this service account to read the queue length, and your job will use it to read messages from the queue:
SERVICE_ACCOUNT_NAME=gcppubsubtest
PROJECT_ID=$(gcloud config get-value project)
SERVICE_ACCOUNT_FULL_NAME=$SERVICE_ACCOUNT_NAME@$PROJECT_ID.iam.gserviceaccount.com
gcloud beta iam service-accounts create $SERVICE_ACCOUNT_NAME \
--display-name "PubSub Sample"gcloud iam service-accounts keys create $(pwd)/sa.json \
--iam-account $SERVICE_ACCOUNT_FULL_NAMEgcloud projects add-iam-policy-binding $PROJECT_ID \
--member serviceAccount:$SERVICE_ACCOUNT_FULL_NAME \
--role roles/monitoring.viewer
Next we will create the topic and the subscription, and setup permissions:
SUBSCRIPTION_NAME=mysubscription
TOPIC_NAME=mytopic# Create Topic
gcloud beta pubsub topics create $TOPIC_NAME# Create Subscription
gcloud beta pubsub subscriptions create $SUBSCRIPTION_NAME \
--topic $TOPIC_NAME# We need to give the service account access to the subscriber to
# receive messages
gcloud beta pubsub subscriptions add-iam-policy-binding $SUBSCRIPTION_NAME \
--member=serviceAccount:$SERVICE_ACCOUNT_FULL_NAME \
--role=roles/pubsub.subscriber
Copy the following code and save it as scaledjob_v.yaml:
Then, we create a secret with the service account, and deploy the ScaledJob. We create these in a new namespace so they are easier to clean up later.
kubectl create ns keda-pubsub-test
kubectl create secret generic pubsub-secret \
--from-file=GOOGLE_APPLICATION_CREDENTIALS_JSON=./sa.json \
--from-literal=PROJECT_ID=$PROJECT_ID \
-n keda-pubsub-test# Deploy a ScaledJob
kubectl apply -f scaledjob_v2.yaml
Validate that the ScaledJob was created and print the KEDA operator logs for debug purposes:
kubectl -n keda-pubsub-test get scaledjob# Output:
# NAME TRIGGERS ....
# test-keda-pubsub-scaledjob gcp-pubsub ...kubectl -n keda logs Deployment/keda-operator
Finally, we can send a message to PubSub and validate that a pod was created to handle the message:
# Send a message to pubsub
gcloud pubsub topics publish $TOPIC_NAME --message="test123"# Check if any pods were created
$ kubectl -n keda-pubsub-test get poNAME READY STATUS RESTARTS AGE
test-keda-pubsub-scaledjob-4wkq7-bbvhj 0/1 Completed 0 38m# Print the pod’s logs
$ kubectl -n keda-pubsub-test logs test-keda-pubsub-scaledjob-4wkq7-bbvhjWaiting for a message for 30 seconds
Received: b'test123'.Received and acknowledged 1 messages from projects/j.../subscriptions/mysubscription.
Tips for working with GCP PubSub
- Set read timeout. KEDA’s PubSub scaler relies on StackDriver metrics. These metrics may take several seconds to update, causing KEDA to spin off new jobs until the queue length is updated. To mitigate the issue, make sure to set a time limit when reading from PubSub, as I did in the example above
- Extending ack deadline. By default, a PubSub client needs to acknowledge the message within 10 seconds, otherwise PubSub resends the message. If you are reading this guide, your jobs probably take more than 10 seconds, so you should keep extending the acknowledgement deadline. For an example, check out synchronous_pull_with_lease_management() in Google’s examples — https://github.com/googleapis/python-pubsub/blob/master/samples/snippets/subscriber.py
Special thanks to the support from the KEDA community in Slack, and especially Tsuyoshi Ushio who helped fixing a couple of issues I found in the preview version of KEDA 2.
