MongoDB-Powered Autoscaling: Harnessing KEDA to Scale Applications Dynamically Based on Database Events Triggered by MongoDB Query Results.

Intoduction:

In the realm of dynamic application scaling, Kubernetes Event-Driven Autoscaling (KEDA) emerges as a game-changer, offering a versatile platform for efficient resource management.

MongoDB, renowned for its flexibility and scalability, seamlessly integrates with KEDA to empower developers with advanced scaling capabilities.

Empowering Dynamic Autoscaling with MongoDB and KEDA

In this blog, we’re poised to unveil a comprehensive, step-by-step guide on setting up MongoDB query-based autoscaling, empowering you to optimize resource allocation seamlessly and efficiently

Imagine an application faced with the daunting task of validating thousands of rows within user-uploaded CSV or Excel files.However, relying solely on the server to perform these tasks within the same service might overload the server, potentially causing performance issues.

So, what’s the solution? Enter microservices with autoscaling capabilities, tailored to activate only when validation tasks arise. Leveraging event-driven autoscaling based on MongoDB events, you can establish a scalable architecture.

Here’s how it works: When a user uploads a file, a MongoDB document with the for ex. with status ‘validating’ is created in the database. This document, with its designated status, serves as the trigger for scaling your application. You can customize the query for scaling the application in the configuration file to suit your specific task, similar to the scenario mentioned above. the application dynamically scales up to handle the validation process efficiently. Once the validation is complete and the status of the MongoDB document changes, indicating success or failure, the trigger becomes inactive, and the application gracefully scales down.

During periods when no trigger is active, the service will remain at zero replicas, effectively conserving resources and minimising costs.

In this way, event-driven autoscaling based on MongoDB events offers a streamlined approach to managing resource allocation, ensuring optimal performance while mitigating server strain.

Let’s get started with the setup.

First, let’s set up everything locally. The prerequisite is to have Docker installed on your machine. Once installed, open your Docker Desktop and enable Kubernetes from there.

Ensure that KEDA is deployed in your local environment. You can refer to the official website for deploying KEDA: https://keda.sh/docs/2.13/deploy/

Next, create your application images. We will consider two images for example for two services: one for the backend server and one for the autoscaling app, which will run on trigger.

You can use the following simple image build templates for a Node.js application to build images locally.

FROM node

RUN mkdir -p /home/app

ENV MONGO_DB_USERNAME=username \
    MONGO_DB_PWD=password

COPY ./app /home/app

CMD ["node","/home/app/src/index.js"]

This is a sample template to create your Node.js Application images. You can modify the templates and environment variables as per the requirements of the project.

Now that you have both Dockerfiles for your server app and autoscaling app,

Let’s create two images. We’ll name the images ‘app:1.0’ for the server and ‘kapp:1.0’ for our autoscaling app.

Use the following commands to build the images locally. Ensure that your application folder and Dockerfile are at the same directory level

docker build -t app:1.0 .
docker build -t kapp:1.0 .

Now, let’s quickly write all the Kubernetes YAML configuration files.

1. Deployment YAML file for the app(app.yml):

In this configuration, we designate the container name as “app” and set the app service as a LoadBalancer, allowing it to be accessible outside the Kubernetes cluster, exposing it on port 5000.

For example, let’s consider app as simple application where there is a server with an endpoint to create a document in MongoDB with the status as “validating.”

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  namespace: default
  labels: 
    app: app
spec:
  replicas: 1
  selector: 
    matchLabels:
      app: app
  template: 
    metadata:
      labels:
        app: app

    spec: 
      containers:
      - name: app
        image: app:1.0
        ports:
        - containerPort: 3000
        env: 
          - name: MONGO_PASSWORD
            valueFrom:
              secretKeyRef:
                name: mongodb-secret
                key: password

---

apiVersion: v1
kind: Service
metadata: 
  name: app-service
spec:
  selector:
    app: app
  type: LoadBalancer
  ports:
  - protocol: TCP
    port: 5000
    targetPort: 3000
    nodePort: 32000

2. Deployment YAML file for kapp(kapp.yml):

In this configuration, we designate the container name as “kapp” and set the kapp service as a LoadBalancer, enabling it to be accessible outside the Kubernetes cluster, with exposure on port 4000. Depending on your application, you may choose to access or restrict the access.

For example, let’s consider a simple scenario for kapp where the code will execute to check the database for a document with the status as “validating” and update it to “success”.

This kapp will be triggered and scale up when the status is “validating”, and when it changes the status to “success”, the trigger will become inactive and scale down.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kapp
  namespace: default
  labels: 
    app: kapp
spec:
  replicas: 1
  selector: 
    matchLabels:
      app: kapp
  template: 
    metadata:
      labels:
        app: kapp

    spec: 
      containers:
      - name: kapp
        image: kapp:1.0
        ports:
        - containerPort: 4000
        env: 
          - name: MONGO_PASSWORD
            valueFrom:
              secretKeyRef:
                name: mongodb-secret
                key: password

---

apiVersion: v1
kind: Service
metadata: 
  name: kapp-service
spec:
  selector:
    app: kapp
  type: LoadBalancer
  ports:
  - protocol: TCP
    port: 6000
    targetPort: 4000
    nodePort: 32100

3. Secret YML file (mongo-secret.yml)

This file will contain all the secrets needed for your application as environment variables (env).

We will use a MongoDB connection string to connect to MongoDB. You can use MongoDB Atlas for this purpose.

store mongoDB string in base64 format in the connect variable

apiVersion: v1
kind: Secret
metadata: 
  name: mongodb-secret
type: Opaque
data: 
  username: dXNlcm5hbWU=
  password: cGFzc3dvcmQ=
  connect: bW9uZ29kYjovL3VzZXJuYW1lOnBhc3N3b3JkQG1vbmdvZGItc2VydmljZS5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsOjI3MDE3L3VzZXItYWNjb3VudD9hdXRoU291cmNlPWFkbWlu

you can use simple bash command to convert your mongo string to base64 format.

echo -n <your mongo string> | base64

4. ScaledObject configuration YML file (keda.yml):

This file will contain the configuration for KEDA. In this file, we have given a name to this ScaledObject as ‘mongodb-job’. We have set ‘scaleTargetRef’ to ‘kapp’, so it will scale up the container named ‘kapp’ (second configuration which we have written).

We have defined the type of trigger to be used as a MongoDB trigger. Then, we have provided the details of the database and collection to be monitored. Additionally, we have specified a query as ‘{ “status”: “validating” }’. With ‘queryValue’ set to 1, it will scale the ‘kapp’ pod from 0 to 1 replica set when the query returns 1 result. Here, we have set the cooldown period to 30 seconds, so it will check every 30 seconds.”

We have also defined TriggerAuthentication in this file, which will use the MongoDB string stored in the secret.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: mongodb-job
  namespace: default
spec:
  scaleTargetRef:
    name: kapp
  cooldownPeriod:  30

  triggers:
  - type: mongodb
    metadata:
      dbName: "example_app"
      collection: "bulk_data"
      query: '{ "status": "validating" }'
      queryValue: "1"
    authenticationRef: 
      name: mongodb-trigger
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata: 
  name: mongodb-trigger
spec: 
  secretTargetRef:
    - parameter: connectionString
      name: mongodb-secret
      key: connect

Now that we have all the configurations, let’s start the cluster by running the following commands. Make sure to execute these terminal commands in the folder where all the configuration files are stored.

kubectl apply -f mongo-secret.yaml
kubectl apply -f app.yaml
kubectl apply -f kapp.yaml
kubectl apply -f keda.yaml

After executing these commands, check that all deployments are running correctly with the following command

"kubectl get pods"

You will see status of app running.

However, you may notice that ‘kapp’ is not running. This is because the ‘keda.yml’ file, when applied, has deactivated ‘kapp’.

You can confirm this by running the command:

kubectl describe so

you will see following event. “Deactivated apps/v1.Deployment default/kapp from 1 to 0”

with message as “Scaling is not performed because triggers are not active”

Let’s use the app server to create a document with the status as “validating”. You will observe the event as “Scaled apps/v1.Deployment default/kapp from 0 to 1” with the message: “Scaling is performed because triggers are active” by running the same kubectl describe so command.

So, here our kapp service got scaled up and performed the task, updating the status as “success” in the database. This action caused the trigger to become inactive, resulting in it getting scaled down to 0.

And there you have it! You can now utilize MongoDB-based autoscaling to tailor-fit your application’s needs.

If you have any questions, feel free to ask!

Thank you…..