Virtual Machine Orchestration in Kubernetes using KubeVirt

Introduction

At first, it seems very odd and confusing to think about Virtual Machine orchestration using Kubernetes. Thinking about how a VM can be provisioned inside a Kubernetes cluster. Whether it will run inside pods? or along side a pod? It’s confusing and seem to be not possible. But, I will have to break it to you that it’s quite possible and easy to configure with the use of KubeVirt.

KubeVirt is an open-source sandbox project of CNCF which is a Kubernetes add-on that provides users the ability to schedule traditional virtual machine workloads side by side with container workloads. Through the use of Custom Resource Definitions (CRDs) and other Kubernetes features, KubeVirt seamlessly extends existing Kubernetes clusters to provide a set of virtualization APIs that can be used to manage virtual machines.

KubeVirt technology addresses the needs of development teams that have adopted or want to adopt Kubernetes but possess existing Virtual Machine-based workloads that cannot be easily containerized. More specifically, the technology provides a unified development platform where developers can build, modify, and deploy applications residing in both Application Containers as well as Virtual Machines in a common, shared environment.Benefits are broad and significant. Teams with a reliance on existing virtual machine-based workloads are empowered to rapidly containerize applications. With virtualized workloads placed directly in development workflows, teams can decompose them over time while still leveraging remaining virtualized components as is comfortably desired.

Offering a unified API managing regular Kubernetes workloads, and virtual machines. Kubernetes clients would be able to manage virtual disks and qemu/kvm virtual machines, while being subject to Kubernetes RBAC, quotas, through a well-known API, etc. Compared to other virtualization tools, Kubevirt doesn’t require Kubernetes end-users being able to log into vmware/vsphere.

Architecture

Here in this architecture, it’s a typical Kubernetes architecture where we have a Master node, two worker node and three namespaces. Inside the namespaces, just like how pods are running, VMs are also running. By using specific CRD’s from KubeVirt, we are able to provision and manage VMs within a Kubernetes cluster.

You can manage VMs in a Kubernetes cluster using KubeVirt if it’s an on-prem setup and nested-virtualization is enabled in the worker nodes. However, if you want to do the same in a cloud platform like AWS, ordinary VM instances are not enough. You should be using ‘metal’ instances like ‘m5.metal’ in AWS.

Next, we will explore how we can configure and setup VMs in a Kubernetes cluster using KubeVirt.

Prerequisites

Server provisioning:
Three servers need to be provisioned. One for the master node with atleast 2GB vCPU and 8GiB of memory, another one for the worker node(requires BareMetal VMs provisioned using Type1 Hypervisors). If required to set the Kubernetes cluster in any public cloud, the worker node should be a metal instance. Eg: m5.metal instance type. Also, make sure to add additional EBS volume (Min 20 GiB of storage Gp2) if you are using AWS cloud.

Kubernetes Master node:
A Kubernetes master node needs to be created using ‘Kubeadm’ and any CNI like Antrea, Calico or Flannel.

Kubernetes Worker node:
A Kubernetes worker node need to be created using ‘Kubeadm’.

If you have trouble setting up a Kubernetes cluster using Kubeadm, view this article that clearly instructs how to do so. Click here

Configurations

KubeVirt Setup

1. After setting up the Master node and worker node in the Kubernetes cluster and setting up the CNI, run the following commands to install the KubeVirt operators and CRDs in the Kubernetes cluster.

export VERSION=$(curl -s https://api.github.com/repos/kubevirt/kubevirt/releases | grep tag_name | grep -v -- '-rc' | sort -r | head -1 | awk -F': ' '{print $2}' | sed 's/,//' | xargs)
echo $VERSION
wget https://github.com/kubevirt/kubevirt/releases/download/${VERSION}/kubevirt-operator.yaml
wget https://github.com/kubevirt/kubevirt/releases/download/${VERSION}/kubevirt-cr.yaml
kubectl create -f kubevirt-operator.yaml
kubectl create -f kubevirt-cr.yaml

2. Run the following command to check whether the KubeVirt CRDs and Operators have been created and running in the cluster.

kubectl get all -n kubevirt

The above commands will help setup KubeVirt in the cluster.

VirtCtl Setup

KubeVirt provides an additional binary called virtctl for quick access to the serial and graphical ports of a VM and also handle start/stop operations. Go through the following steps to install virtctl. virtctl is used to start, stop and delete the VMs provisioned using KubeVirt.
(Refer: https://github.com/kubevirt/kubevirt.github.io/blob/main/_includes/quickstarts/virtctl.md if you encounter any issues)

1. Run the following steps. (Select and run all in once)

VERSION=$(kubectl get kubevirt.kubevirt.io/kubevirt -n kubevirt -o=jsonpath="{.status.observedKubeVirtVersion}")
ARCH=$(uname -s | tr A-Z a-z)-$(uname -m | sed 's/x86_64/amd64/') || windows-amd64.exe
echo ${ARCH}
curl -L -o virtctl https://github.com/kubevirt/kubevirt/releases/download/${VERSION}/virtctl-${VERSION}-${ARCH}
chmod +x virtctl
sudo install virtctl /usr/local/bin

2. Run the following to check whether VirtCtl has been properly installed.

virtctl

VM Provisioning using Cirros based demo OS

This will download a VM manifest and running it will create VirtualMachineInstance in the Cluster. After that, we need to start the VM using virtctl.

1. Download the manifest file.

wget https://kubevirt.io/labs/manifests/vm.yaml

2. Create the VM using the manifest file. Note that the image used is cirros based which is a base image for checking network connectivity, etc. It doesnt have any package mananger or anything installed.

kubectl apply -f https://kubevirt.io/labs/manifests/vm.yaml

3. Run the following command to list the VM.

kubectl get vms

4. Run the follwing command to start the VMI. The VM and VMI should be in running phase after running this command.

virtctl start testvm

5. Now that the Virtual Machine Instance has been started, check the status. Check if the VMI is in running status. Also, a pod(VM launcher which runs in 2/2) will also be created in the default namespace. Check if the pod is in running state. If not, we will have to troubleshoot it.

kubectl get vmi
kubectl get pods -n default | grep -i launcher

6. If the VM is in running state and to access it, run the following command.

virtctl console --kubeconfig=$KUBECONFIG testvm

Now you have to enter the user login and password as suggested by the cirros VM login info. Once the credentials are validated, you will login to the VM.

Deploying Virtual Machine Instance using Fedora container-disk

Here we will deploy a VM based of Fedora container-disk image using KubeVirt.

1. Create a YAML file and copy the following contents into the YAML file.

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: fedora-vmi
spec:
  running: false
  template:
    metadata:
      labels:
        kubevirt.io/size: small
        kubevirt.io/domain: fedora-vmi
    spec:
      domain:
        devices:
          disks:
            - name: containerdisk
              disk:
                bus: virtio
            - name: cloudinitdisk
              disk:
                bus: virtio
          interfaces:
          - name: default
            masquerade: {}
        resources:
          requests:
            memory: 1Gi
          limits:
            memory: 2Gi
      networks:
      - name: default
        pod: {}
      volumes:
        - name: containerdisk
          containerDisk:
            image: kubevirt/fedora-cloud-container-disk-demo:latest
        - name: cloudinitdisk
          cloudInitNoCloud:
            userData: |-
              #cloud-config
              password: fedora
              chpasswd: { expire: False }

Run the following command to create the VM instance.

kubectl create -f <file name>

2. Run the following command to see the Virtual Machine.

kubectl get vm

Now we need to start the Virtual Machine Instance. Run the following command.

virtctl start fedora-vmi

3. Now that the Virtual Machine Instance has been started, check the status. Check if the VMI is in running status. Also, a pod(VM launcher which runs in 2/2) will also be created in the default namespace. Check if the pod is in running state. If not, we will have to troubleshoot it.

kubectl get vmi
kubectl get pods -n default | grep -i launcher

**Note:** In the case of hosting applications in the VM and to expose the VM to the outside world, you can create a service by running the following command.

virtctl expose virtualmachineinstance fedora-vmi --name fedora-vmi-service-port --port <application-port> --target-port <application-port> --type NodePort

4. Use the following command to login to the Fedora VM.

virtctl console - kubeconfig=$KUBECONFIG fedora-vmi

The boot load sequence will run and complete. Then it will ask for credentials.

• *Note:** Note that the default username is ‘fedora’ and password is also ‘fedora’ since that’s what we have set by using the userdata in the YAML file.

Creating custom container-disk

Here, we will learn how to create a custom container-disk using a Dockerfile, pushing it to a container registry and then provisioning a Ubuntu based VM using KubeVirt.

A container disk is a virtual machine image that is stored as a container image in a container image registry. You can use container disks to deliver the same disk images to multiple virtual machines and to create large numbers of virtual machine clones. The containerDisk feature provides the ability to store and distribute VM disks in the container image registry. containerDisks can be assigned to VMs in the disks section of the VirtualMachineInstance spec.

containerDisks are ephemeral storage devices that can be assigned to any number of active VirtualMachineInstances. This makes them an ideal tool for users who want to replicate a large number of VM workloads that do not require persistent data. containerDisks are commonly used in conjunction with VirtualMachineInstanceReplicaSets.No network shared storage devices are utilized by containerDisks. The disks are pulled from the container registry and reside on the local node hosting the VMs that consume the disks.

1. First go to any Distro cloud image site and download a Disk Image File. It should be of extension ‘.qcow2c’. In this scenario, I chose the ‘CentOS-7-x86_64-GenericCloud-2111.qcow2c’.

Ubuntu: https://cloud-images.ubuntu.com/
Centos7: https://cloud.centos.org/centos/7/images/

2. Next create a Dockerfile based on Ubuntu.

FROM scratch
ADD --chown=107:107 CentOS-7-x86_64-GenericCloud-2111.qcow2c /disk/

3. Run the Docker build command to build the image. In this case, we chose Docker Hub as the container registry.

docker build -f centos-container-disk-dockerfile -t <account>/centos-container-disk:latest .

4. Once the image is finished building, we need to push it to the container registry. Run the following command to push it to the container registry.

docker push <account>/centos-container-disk:v1

5. Now we can point to this image when we write a YAML file for VirtualMachine Instances based on KubeVirt.

apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachineInstance
metadata:
  name: centos-container-disk
spec:
  domain:
    resources:
      requests:
        memory: 2Gi
      limits:
        memory: 3Gi
    devices:
      disks:
      - name: containerdisk
        disk: {}
      - disk:
          bus: virtio
        name: cloudinitdisk
  volumes:
    - name: containerdisk
      containerDisk:
        image: arbabu/centos-container-disk:latest
        path: /disk/CentOS-7-x86_64-GenericCloud-2111.qcow2c
    - name: cloudinitdisk
      cloudInitNoCloud:
        userData: |-
          password: centos
          chpasswd: { expire: False }

6. Now create the Virtual Machine Instance and test out whether the VMI gets booted and loaded correctly. Since we are directly creating a VirtualMachineInstance, there’s no need to use the ‘virtctl start’ command to start any VirtualMachine.

kubectl create -f centos-container-disk

7. Now, login to the VMI by running the following command.

virtctl console - kubeconfig=$KUBECONFIG centos-container-disk

The boot load sequence will run and complete. Then it will ask for credentials. Then login to the VM.

You can try the same with any other Linux distros that offer cloud-images of type ‘.qcow2c’.

Why KubeVirt?

When you are maintaining different or multiple infrastructures for your containerised workload and VM workloads then you maintain separate layers of logging, monitoring, metrics, scheduling capabilities and networking.

With KubeVirt you get to leverage the power of Kubernetes to run VM workloads alongside container workloads and get the same benefits including:

1. Declarative: Use the declarative approach to create a VM by creating a custom resource of kind VirtualMachine. This gives you have similar experience of creating a pod, deployment or any other Kubernetes object.
2. Orchestration: You get the benefits of Kubernetes: the scheduling capabilities, resource requests and limits, RBAC, service accounts and the same networking as you have for the containers.
3. Storage: You can declare and use PersistentVolumeClaims (PVC) as disks inside the VM.
4. Observability: You can use the same observability tools for your VM workloads as you have for your containerised workloads. You can have the same systems for logging, allowing you to create an integrated dashboard and alerting.
5. Same infrastructure: Since the VM and the container workloads will be running side by side, you need not maintain two separate infrastructures.
6. Pipelines: You can leverage different tools like GitLab, ArgoCD to configure release pipelines that can deploy both containers and VMs in a single go.

Resources and References

Installation - KubeVirt User-Guide

Disks and Volumes - KubeVirt User-Guide

Virtual Machines Instances - KubeVirt User-Guide

Accessing Virtual Machines - KubeVirt User-Guide

kubevirt/container-disk-images.md at main · kubevirt/kubevirt

[never touched kubevirt?] 15 minute quick start kubevirt - Develop Paper