Using VMSS Flex in Self-Managed Kubernetes Clusters on Azure

A deep dive into the Azure VMSS Flex based solution for self-managed Kubernetes cluster

7 min readMay 22, 2023

In January 2023, Microsoft Azure announced the Public Preview of VMSS Flex in self-managed Kubernetes clusters. This feature unblocks customers from leveraging the latest features that VMSS Flex provides to embrace higher reliability and better performance in large Kubernetes clusters. In this blog, we will take a close look at this new feature on Azure and deploy a self-managed Kubernetes cluster using VMSS Flex step by step.

This blog covers the blow aspects:

Cluster provisioning
CNI installation
Out-of-tree Cloud Controller Manager and Cloud Node Manager installation
Azure Disk Container Storage Interface (CSI) Driver Installation
Workload deployment

1. Cluster Provisioning

The Kubernetes community provides many tools to provision self-managed Kubernetes cluster on Azure (for example, CAPZ, Kubespray, terraform, etc.). You can select any of these cluster provisioning tools as long as the tool you select supports provisioning VMSS Flex VM as Kubernetes nodes. One example of these tools is Cluster API Provider Azure (CAPZ), of which the version 1.7 has the official support of provisioning self-managed Kubernetes clusters with VMSS Flex. You can follow the CAPZ doc to learn more about CAPZ if you are interested in. However, the following steps are generic no matter which tool you choose for the cluster provisioning.

2. CNI Installation

The self-managed Kubernetes cluster is not ready for use yet after provisioned. If you query the node status of the cluster, you will find all the nodes are in “NotReady” status:

Nodes are “NotReady” yet as the CNI is not installed

This is expected as the network plugin of the cluster is not set up yet. There are multiple different CNI solutions in public. We will use Calico VXLAN as an example in the blog. To install Calico VXLAN CNI:

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-api-provider-azure/main/templates/addons/calico.yaml

You should be able to see the nodes become “Ready” gradually after the CNI is installed.

Nodes become “Ready” after CNI installation

In this next step, we will install the out-of-tree Cloud Controller Manager and Cloud Node Manager to enable Azure specific control plane functions.

3. Out-of-tree Cloud Controller Manager and Cloud Node Manager Installation

Cloud Controller Manager (CCM) and Cloud Node Manager (CNM) are the Azure out-of-tree implementations of the control plane components which embeds the cloud specific logic. These two components unlock a wide set of azure specific features such as node address & zone discovery, cloud load balancers, IP address management, and cluster networking via VPC routing tables. In December 2022, Azure published the release of CCM and CNM version 1.26.0, which for the first time, officially support managing VMSS Flex VMs as Kubernetes nodes. The latest version of CCM and CNM available at the time this blog is authored is v1.26.4, but please feel free to use any newer version if it is available.

The easiest way to install CCM and CNM is with Helm, a popular across platform package manager for Kubernetes:

export CCM_IMAGE_REGISTRY=mcr.microsoft.com/oss/kubernetes 
export CNM_IMAGE_REGISTRY=mcr.microsoft.com/oss/kubernetes 
export CCM_IMAGE_NAME=azure-cloud-controller-manager 
export CNM_IMAGE_NAME=azure-cloud-node-manager 
export CCM_IMAGE_TAG=v1.26.4 
export CNM_IMAGE_TAG=v1.26.4 
export CLUSTER_NAME=your_cluster_name

 helm install --repo https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/helm/repo cloud-provider-azure --generate-name --set infra.clusterName="${CLUSTER_NAME}" --set cloudControllerManager.imageRepository="${CCM_IMAGE_REGISTRY}" --set cloudNodeManager.imageRepository="${CNM_IMAGE_REGISTRY}" --set cloudControllerManager.imageName="${CCM_IMAGE_NAME}"  --set cloudNodeManager.imageName="${CNM_IMAGE_NAME}" --set-string cloudControllerManager.imageTag="${CCM_IMAGE_TAG}" --set-string cloudNodeManager.imageTag="${CNM_IMAGE_TAG}"

The output of the above command should be like this:

With the above command, a Deployment named cloud-controller-manager is installed into one of the control plane nodes, and a DaemonSet named cloud-node-manager is deployed on each node. You can monitor the status of these control plane components by running the below commands:

kubectl get Deployment cloud-controller-manager -n kube-system 

kubectl get DaemonSet cloud-node-manager -n kube-system

After both CCM and CNM are ready, the self-managed Kubernetes cluster is ready to deploy stateless workloads.

4. Azure Disk Container Storage Interface (CSI) Driver Installation

Azure Disk Container Storage Interface (CSI) Driver is required if you want to deploy stateful workloads such as StatefulSet which requires access of Azure Disk volume. Azure Disk CSI Driver v.125.0 was released in November 2022 and is the first version to support attach/detach data disks on VMSS Flex nodes in self-managed Kubernetes cluster. There are two steps to install the CSI Driver:

Step 1: Define the StorageClass
Step 2: Deploy the CSI Disk Controller

4.1 StorageClass Creation

StorageClass defines the disk parameters such as SKU as well as volume binding policies. The list of available configurations can be found in the Kubernetes official document. Here we will use StandardSSD_LRS and bind the volume when it is to be consumed as an example. To do that, we first create a yaml file and name it as storageclass.yaml. Then, put the blow contents into the file just created:

--- 

apiVersion: storage.k8s.io/v1 
kind: StorageClass 
metadata: 
  name: managed-csi 
provisioner: disk.csi.azure.com 
parameters: 
  skuName: StandardSSD_LRS 
reclaimPolicy: Delete 
volumeBindingMode: WaitForFirstConsumer 
allowVolumeExpansion: true

We can submit this StorageClass definition to the Kubernetes cluster with the below command:

kubectl create -f storageclass.yaml

StorageClass creation

Now the StorageClass is ready to be used by the CSI Driver.

4.2 CSI Disk Controller Deployment

Similar to CCM and CNM installation, we can use helm to install the CSI Driver:

helm repo add azuredisk-csi-driver https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/master/charts
helm repo update azuredisk-csi-driver
helm search repo -l azuredisk-csi-driver
helm install azuredisk-csi-driver azuredisk-csi-driver/azuredisk-csi-driver --namespace kube-system

The output of the above command should be like this:

You can run the below commands to further verify the CSI Driver installation results:

kubectl get Deployment csi-azuredisk-controller -n kube-system

Output:

kubectl get Daemonset csi-azuredisk-node -n kube-system

Output:

Up to now, we have installed all the required control plane components and your self-managed Kubernetes cluster is fully functional and ready for use!

5. Workload Deployment

It’s time to deploy a workload to verify the functions of this self-managed Kubernetes cluster. In this blog, we will create an external load balancer service with a StatefulSet of Nginx pods behind it. This workload can validate a list of Azure specific features including node address & zone discovery, cloud load balancers, IP address management, network routing with route table, data disk attach.

Create a file with name elb-statefulset.yaml and put the below contents into it:

---
apiVersion: v1
kind: Service
metadata:
  name: elb
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nginx-statefulset
  labels:
    app: nginx
spec:
  podManagementPolicy: Parallel  # default is OrderedReady
  serviceName: elb
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      nodeSelector:
        "kubernetes.io/os": linux
      containers:
        - name: nginx-statefulset
          image: mcr.microsoft.com/oss/nginx/nginx:1.19.5
          volumeMounts:
            - name: persistent-storage
              mountPath: /mnt/azuredisk
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: nginx
  volumeClaimTemplates:
    - metadata:
        name: persistent-storage
        annotations:
          volume.beta.kubernetes.io/storage-class: managed-csi
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi

Submit this file to the Kubernetes cluster with the below command:

kubectl apply -f elb-statefulset.yaml

Apply elb-statefulset.yaml

You should see an external load balancer is created with name elb and a StatefulSet with 2 replicas:

kubectl get service elb

kubectl get StatefulSet nginx-statefulset

Outputs of external load balancer and StatefulSet

2 data disks are created in the resource group and attached to the VMSS Flex nodes. You can also verify the data plane traffic of this workload using public ip of the load balancer.

Verify the data plane connection with the external load balancer created

6. Summary

In this blog, we demonstrated how to deploy a self-managed Kubernetes cluster with VMSS Flex on Azure step by step. This feature is available for any self-hosted K8s solutions on Azure, and is compatible with AKS and will be adopted by AKS as Up Stream.

We have performed scale and performance testing to validate the reliability of this feature in large Kubernetes clusters. We also quantitatively measured the performance improvements using VMSS Flex in comparison with VMSS Uniform with A/B Testing in the scenarios of scaling up / down, StatefulSet Disk Attach / Detach, Internal / External Load Balancer with backend pods.

Currently, Azure Managed Kubernetes Cluster (AKS) does not support using VMSS Flex and agent pool yet, but we would expect to see this feature in AKS soon.

Please feel free to create your own self-managed Kubernetes cluster on Azure with VMSS Flex and let me know your comments.