Building End-to-End MLOps Pipelines for Sentiment Analysis on Azure with Terraform, Kubeflow v2, Mlflow, and Seldon: Part 3

8 min readJun 13, 2023

Part 1: Introduction and Architecture Planning

Part 2: Developing Workflows for Infrastructure Deployment via CI/CD

Part 4: Setting up Kubeflow and Seldon on AKS

Part 5: End-to-End Training Pipeline and Inference Deployment using Seldon.

In Part 3 we are going to going to install Mlflow and it’s dependent resources inside our AKS cluster that we provisioned in Part 2. In order to deploy MLflow we need to deploy a Database as a prerequisite so for this implementation we will deploy Postgres in our AKS cluster and then use it as our default backend storage solution. All the manifests used here are available on: https://github.com/rahuja23/MLFlow_Deployment.

Deploying PostgresSQL on Kubernetes

In order to deploy PostgreSQL inside our Kubernetes cluster following steps needs to be followed sequentially:

Creation of Namespace

In order to manage and maintain all our resources related to PostgreSQL we create a namespace and we will provision all our resources inside this namespace. In order to create the namespace we need to run:

kubectl create ns postgres

Creation of Persistence Volume and Persistence Volume Claim

Since Kubernetes basics are not in the scope of this article, I would highly suggest readers to read the official documentation of Kubernetes in order to understand what Persistence Volume (PV) and Persistence Volume Claims (PVC) are. In order to deploy stateful applications such as PostgreSQL database we’ll have to create a PVC to store the database data. In this implementation we will create a pod that mounts this PVC and runs MySQL database.

For this tutorial, we will move forward with a local volume using ‘mnt/datavol’ as the path to the volume:

cat <<EOF > postgres-pv.yaml

kind: PersistentVolume
apiVersion: v1
metadata:
  name: postgres-pv-volume  # Sets PV's name
  labels:
    type: local  # Sets PV's type to local
    app: postgres
spec:
  storageClassName: manual
  capacity:
    storage: 5Gi # Sets PV Volume
  accessModes:
    - ReadWriteMany
  hostPath:
    path: "/mnt/datavol"
EOF

cat <<EOF > postgres-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: postgres-pv-claim  # Sets name of PVC
  labels:
    app: postgres
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteMany  # Sets read and write access
  resources:
    requests:
      storage: 5Gi  # Sets volume size
EOF

# Applying the manifests for the PV 
kubectl create -f postgres-pv.yaml
# Applying the manifests for the PVC
kubectl create -f postgres-pvc.yaml

Now this should create our Persistent Volume and it’s corresponding Persistence Volume Claim. In order to check this we can run the following commands:

Persistence Volume and Persistence Volume Claim

Create and apply ConfigMap

Configmaps are used in order to prevent secrets like: credentials etc of inside our application code from exposing inside the source code. It alo helps the developers from separating data from code. With the help of Configmaps one can easily deploy and update the data used by the application without actually changing the source code.

A configmap can be easily created by copy pasting the following code snippet:

cat <<EOF > postgres-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-config
  labels:
    app: postgres
  namespace: postgres
data:
  POSTGRES_DB: postgresdb
  POSTGRES_USER: **USERNAME**
  POSTGRES_PASSWORD: **PASSWORD**

EOF

# Applying the manifests
kubectl create -f postgres-config.yaml

Now we can check whether the Configmap has been created or not:

Configmap used by PostgreSQL

Create and apply PostgreSQL as a Deployment

Deployments in Kubernetes is a declarative way in which one can define how an application should be deployed or updated inside the cluster. They are a way to manage the roll out and update process of an application inside a Kubernetes cluster.

cat <<EOF > postgres-dep.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres  # Sets Deployment name
  namespace: postgres
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:10.1 
          imagePullPolicy: "IfNotPresent"
          ports:
            - containerPort: 5432  
          envFrom:
            - configMapRef:
                name: postgres-config
          volumeMounts:
            - mountPath: /var/lib/postgresql/data
              name: postgredb
      volumes:
        - name: postgredb
          persistentVolumeClaim:
            claimName: postgres-pv-claim

EOF

# Apply the manifest
kubectl create -f postgres-dep.yaml

We can check the creation of deployment and pods using the following commands:

Pods and Deployment of PostgreSQL

Exposing PostgreSQL Deployment as a Service

Kubernetes Services helps developers exposing their applications in numerous ways. The default method of doing this is using “ClusterIP” service offered by Kubernetes. Cluster IP as the name suggests exposes the service to an address that can only we used inside the cluster itself.

The deployment can be exposed as a service using the following command:

 kubectl expose deployment postgres --type=ClusterIP --name=postgres-service -n postgres

This command exposes our PostgreSQL deployment as a ClusterIP service.

PostgreSQL deployment exposed as a ClusterIP Service

MLFlow Setup

Now that we have our remote storage backend deployed in our Kubernetes cluster we will now go setup Mlflow inside our cluster.

In order to deploy MLflow in our cluster first we will create a minimalist Dockerfile in order to create an image of Mlflow.

cat <<EOF > Dockerfile
From ubuntu:20.04
RUN  apt-get update  && apt install python3-pip libpq-dev -y
WORKDIR /home
COPY . .
RUN pip3 install -r requirements.txt
ENV MLFLOW_SERVER_PORT 5000
EXPOSE $MLFLOW_SERVER_PORT
ENTRYPOINT ["sh", "/home/setup_mlflow.sh"]
EOF

This Dockerfile has a bash file defined as it’s entrypoint which runs our MLFlow server. The next step is to create the bash file which is responsible for execution of our MLFlow server.

#!/bin/bash
BACKEND_STORE_URI="mysql://$MLFLOW_DB_USER:$MLFLOW_DB_PASSWORD@$MLFLOW_DB_HOST:$MLFLOW_DB_PORT/$MLFLOW_DB_NAME"
mlflow server \
    --backend-store-uri "$BACKEND_STORE_URI" \
    --default-artifact-root "$MLFLOW_SERVER_DEFAULT_ARTIFACT_ROOT" \
    --host "$MLFLOW_SERVER_HOST" \
    --port "$MLFLOW_SERVER_PORT" \
    --workers "$MLFLOW_SERVER_WORKERS"

Now the image based on the dockerfile defined above will run a MLflow server but requires a few arguments to be passed as environment variables. Following arguments are required:

- -backed-store-uri: This will be our database that will be used to store and log metrics.
- -default-artifact-rool: This is basically the object storage solution where all the artiacts logged by Mlflow will be stored. In our case we will be using Azure Blog in order to store the metrics. There are a few ways in which we can define how to reach the blob where the artifacts needs to be stored but for this implementation we will be using HTTP method: https://storageaccount.blob.core.windows.net/container/path/to/blob

The image is available on dockerhub as racahu23/mlflow_base:1.

Now again as we did for our PostgreSQL we will provision a namespace where we will deploy all the resources provisioned for MLFlow.

The namespace can be created by running the following command:

kubectl create ns mlflow

Now, we can deploy Mlflow server as a deployment in this namespace. As per the default configuration the server listens on port 5000.

Now we create a configmap where we define all the environment variables which are required for the mlflow server. The manifest will look as follows:

apiVersion: v1
kind: ConfigMap
metadata:
  name: mlflow
  namespace: mlflow
data:
  MLFLOW_DB_HOST: http://postgres-service.postgres.svc.cluster.local
  MLFLOW_DB_USER: admin
  MLFLOW_DB_PORT: "5432"
  MLFLOW_DB_NAME: postgresdb
  MLFLOW_ARTIFACT_URI: "https://storageaccount.blob.core.windows.net/container/path/to/blob"

kubectl create -f configmap.yaml

Configmap containing Env variables for Mlflow Server

In kubernetes one of the most prevalent way to supply passwords or sensitive information used by your deployments is using Secrets. For this implementation the our mlflow deployment requires password for the PostgreSQL deployment which we just created. In order create secret run the following command:

kubectl create secret generic mlflow-db-cred -n mlflow --from-literal=db_password=<PASSWORD>

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mlflow
  namespace: mlflow
  labels:
    app: mlflow

spec:
  replicas: 1
  selector:
    matchLabels:
      app: mlflow
  template:
    metadata:
      labels:
        app: mlflow
    spec:
      containers:
        - name: mlflow
          image: racahu23/mlflow_base:1
          imagePullPolicy: Always
          ports:
            - containerPort: 5000
          envFrom:
            - configMapRef:
                name: mlflow
          env:
            - name: MLFLOW_DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mlflow-db-credent
                  key: db_password
                  optional: false

In the above deployment in order to define the path of our default “BACKEND_STORAGE” we resolved the DNS of the PostgreSQL service that we exposed above. Once the deployment is created the server will start on some node of our AKS cluster. In order to access this server the simplest way is to create a service and port-forward it locally to some port and access it. But in case of production environment we would want the MLflow server to be accessible from outside itself, since we would not like to give access to our kubernetes cluster. So in order to access the Mlflow server from outside we will create a service and ingress rule.

The service can be provisioned using the following script:

apiVersion: v1
kind: Service
metadata:
  name: mlflow
  labels:
    app: mlflow
  namespace: mlflow 
spec:
  selector:
    app: mlflow
  type: ClusterIP
  ports:
    - port: 5000
      targetPort: 5000

The above yaml will create the default “ClusterIP” service of our deployment. To create an ingress rule, we first create an ingress controller which we will install using helm (more details here).

export Namespace='ingress-resources'
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install ingress-nginx ingress-nginx/ingress-nginx --create-namespace --namespace $Namespace --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz

Once the ingress controller is installed, we can have a look at the LoadBalancer which will be provisioned with a public IP to access our MLflow server. In order to get the service following command:

LoadBalancer exposing MLFlow server to a Public IP Address

As it’s shown in the above image helm installed all our ingress resources inside the ingress-resources namespace. It provisioned a LoadBalancer with a Public IP address which can be accessible from anywhere. Only ports 80 and 443 are exposed by Nginx ingress by default. Now, in order to point this public IP to expose Mlflow server we need to create ingress rule for the service of MLflow. The ingress rule is defined as follows:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: mlflow-server-ingress
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  ingressClassName: nginx
  rules:
    - http:
        paths:
          - path: /mlflow(/|$)(.*)
            pathType: Prefix
            backend:
              service:
                name: mlflow
                port:
                  number: 5000

Once the above manifest is applied, the Mlflow server would be publicly accessible. For this demo, Mlflow will be used to log metrics, parameters and model artifacts. The Mlflow server will be publicly accessible on: http://20.13.41.235/mlflow/.