Part 1 of Virtualization on Linux with virsh/QEMU/KVM — What You Need to Install and How

A Very Practical Guide to VMs creation/management on Ubuntu 22.04 LTS

Source: Tenor

This article isn’t aimed at those looking to grasp the basics of virtualization or explore the exciting possibilities virtual machines offer. This article is for those who know why they need it but do not know how to get it started. Many might have had their virtualization experience with the straightforward use of VirtualBox on Windows. However, here I will focus on virtualization on the host Linux OS utilizing QEMU.

Please note: I’m not an experienced system administrator or DevOps engineer; this article is the result of thorough research and experimentation on my part. Should you encounter any inaccuracies or misunderstandings, I encourage you to correct me in the comments. I will very appreciate professional feedback!

This is “Part 1.” That is because I’ve divided the virtualization content I wish to share into 3 segments.
Part 1 will focus on:

• Clarifying the confusion surrounding virtualization tools (hypervisors, emulators, accelerators, etc.) and their names (libvirt, QEMU, KVM, virt-manager)
• Installing ONLY necessary software for virtualization with QEMU and virsh without redundant packages
• Creating the first VM using command line

Part 2 will cover how to create virtual machines using the virsh command line tool, without relying on a graphical interface, and will also cover hardware pass through to created VMs.

Part 3 will look into setting up networks and security measures for VMs.

Key components for running virtualization on Linux OS

Any of you, just by googling “How to create a Virtual Machine in Ubuntu” will almost certainly come across instructions to execute a command like sudo apt install qemu-kvm virt-manager bridge-utils. If blindly executing commands is all you needed, you probably wouldn’t be reading this article: running sudo virt-manager after executing previously mentioned installations will set up virtualization functionality with an easy-to-use GUI, allowing for the creation of numerous VMs with little hustle.

But the fact that you’re reading this suggests you’re looking for rather understanding than just a quick solution. You’re here because you’ve been navigating through a maze of confusing packages/instructions:

and are seeking clarity on what all of these virtualization tools actually do. I’m here to demystify that for you, so let’s start!

The above meme effectively captures the confusion surrounding the virtualization tools available for Linux OS. You might read on forums about one person using virt-manager for virtualization, another using QEMU, a third virtualizing everything with KVM, a fourth opting for libvirt, and yet another creating and executing XML files from the command line.

What are the functions of these tools, and how do they relate to or depend on each other? In order to explain it smoothly I want to start with 3 diagrams that display the common virtualization setups and which core functional components they include:

diagrams created by me

In all 3 setups there is hardware we want to virtualize. A component that is capable of virtualizing host machine’s hardware is known as a hypervisor.

In the Windows setup with VirtualBox (first diagram), VirtualBox is a type 2 hypervisor, which operates as a software layer on a Windows OS. Conversely, KVM in Ubuntu’s Linux Kernel and both Proxmox and VMware are type 1 hypervisors, running directly on the hardware to manage guest VMs.

Very simplified logic of Type 1 Hypervisor: Hardware →Hypervisor of Type 1→VMs

Very simplified logic of Type 2 Hypervisor: Hardware →Host OS →Hypervisor of Type 2 →VMs

Likely, you have come across names like Proxmox and VMware, whose roles in virtualization are illustrated in the 3rd diagram. These are hypervisors of Type 1 and aren’t software packages you can install on your Linux OS for virtualization purposes. For example, Proxmox is a Debian-based Linux distribution with a modified Ubuntu LTS kernel. This means that to utilize Proxmox, you would need to replace your existing OS with it. Both Proxmox and VMware are out of the scope of this article. However, if you’re curious to experiment, you can install one or another on a VM to tinker with 😉.

In this article I focus on a pretty common user setup: having a running OS and wanting to create VMs (displayed on the 2nd diagram). The most popular way in Linux OS to do it is to use QEMU and KVM.
Although KVM is a type 1 hypervisor, it’s particular because it is merged into Linux kernel and this allows to transform a Linux OS into a hypervisor, enabling the running of VMs.

KVM stands for Kernel-based Virtual Machine, it is:

an open source virtualization technology built into Linux®. Specifically, KVM lets you turn Linux into a hypervisor that allows a host machine to run multiple, isolated virtual environments called guests or virtual machines (VMs). (Source: RedHat about virutalization)

QEMU stands for Quick Emulator, it is:

A generic and open source machine emulator and virtualizer. It allows to run operating systems for any machine, on any supported architecture with near native performance. (Source: QEMU.org)

To add some detailed insights about how these two are interconnected:

• Is it possible to virtualize using only KVM? Technically yes, but KVM has neither GUI nor CLI, so one has to write code in C in order to create a VM, but KVM alone will not emulate virtual CPUs or virtual RAM, at the end it is only an API provided by the kernel for user space (source: StackExchange)
• Can you use QEMU without KVM? Yes. QEMU alone can emulate a full system with its built-in binary translator Tiny Code Generator (TCG), but this way involves emulating the CPU, which is compute-intensive task and overall performance of fully emulated system can be slow. Thus, using QEMU without an accelerator is inefficient and generally best for experimental purposes (e.g if your machine is running on amd64 architecture, but you’re curious about exploring how it all works on arm64 architecture). To “accelerate” the emulated system that run on the same architecture to the host’s one QEMU is using accelerators-hypervisors; and KVM is one of them and very popular choice of users. However, QEMU can use alternative accelerators (source: QEMU documentation on accelerators).

After this brief but yet important introduction of inter-connection between QEMU and KVM, let’s proceed with preparing your Linux OS for virtualization. First, you need to verify that your computer is capable of virtualization. Next, you need to install QEMU that will utilize KVM as an accelerator. Lastly, you may want to install tools that will make it easier for you to create and manage VMs.

I step: Check if your machine supports virtualization

It’s essential to understand that not all machines (i.e PCs, laptops) support virtualization; the ability to create and use VMs primarily depends on whether the CPU supports virtualization. To determine if your CPU is capable of supporting virtualization, you can execute the following commands:

lscpu | grep Virtualization
#or
egrep -c '(vmx)' /proc/cpuinfo
#or
kvm-ok

VT-x stands for enabled virtualization functionality for Intel CPU; non-zero output of second command confirms that the system is setup for virtualization; kvm-ok confirms that the system is ready for virtualization

If the outputs of these commands are either blank or zero on you machine, this could mean two things: either your CPU is outdated and incompatible with virtualization technology, or, more likely, virtualization has been turned off in the BIOS settings. You might wonder why it’s turned off by default — is it harmful or does it negatively affect performance? The truth is, it’s not disabled due to performance concerns but rather for security purposes. Although enabling virtualization doesn’t damage your computer or reduce its performance, it is disabled by default to prevent potential malware from exploiting this feature. However, if you’re diligent about safeguarding your computer against malware, you can safely enable and take advantage of this feature (Source: SuperUser).

Why do we only check the CPU for virtualization capabilities and not worry about other parts like the GPU or SSD? It’s because the CPU is the one that does virtualization magic — it abstracts hardware and this allows multiple workloads (VMs) to share a common set of resources(Source: Intel).

II step: QEMU Installation with apt package manager

I’m going to presume that you’ve enabled virtualization and have KVM installed on your system (it comes preinstalled with Linux Kernel version 2.6.0 and later). For KVM to function, it requires an emulator, with QEMU being the most commonly used one.

What makes QEMU particularly appreciated and widely used is its ability to not only simulate various OS but also replicate different hardware architectures. For example, if you have a PC using the amd64 architecture and wish to explore how a system on the arm64 architecture operates, QEMU will be able to make it true for you.

However, there’s a significant consideration to keep in mind. I previously mentioned the ability of QEMU to emulate VMs running on completely different architectures from the host one’s in the context of an experiment. When QEMU does it for you, it cannot use accelerator.

QEMU can make use of KVM when running a target architecture that is the same as the host architecture. For instance, when running qemu-system-x86 on an x86 compatible processor, you can take advantage of the KVM acceleration — giving you benefit for your host and your guest system (QEMU documentation)

If your goal is to utilize QEMU with an accelerator (KVM) for efficient and rapid performance of VMs, with near native performance, you are constrained to emulate VMs only with the same architecture as your host machine’s. If not, QEMU will have to employ for emulation its built-in binary translator, the Tiny Code Generator (TCG), instead of KVM.

Let’s continue with installation of QEMU. Most of the guides for virtualization setup, including the official one from Ubuntu, recommend using the command sudo apt install qemu-kvmto install QEMU. It may appear that this command is required to install all components of QEMU to be used in pair with KVM acceleration. However, this is misleading.

— — — — — — — — — What’s wrong with qemu-kvm?— — — — — — — — — -

QEMU is an open-source project with its own GitHub repository. In the past, a specific version of the project was forked to focus on enhancing QEMU’s performance with the KVM accelerator. But with the release of QEMU version 1.3, all the qemu-kvm fork’s functionalities were merged back with the main branch of qemu-system-x86 (QEMU official page about qemu-kvm fork). Thus, executing sudo apt install qemu-kvm today won’t result in an error; instead, it installs for you qemu-sustem-x86. Proof:

ubuntu@ubuntu:~$ sudo apt install --simulate qemu-kvm
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'qemu-system-x86' instead of 'qemu-kvm'
The following additional packages will be installed:

Is there something bad about it? No. But there is a little “but”. As I mentioned above, QEMU has an impressive capability to emulate VMs with various architectures even with ones that differ from the host’s architecture. However, when you install QEMU with qemu-kvm, your architecture choice to create VMs is limited strictly to those with the x86_64 architecture. If you want to create VMs with different architectures, you should instead install qemu-system.

— – – Difference between qemu-system ,qemu-system-x86 and qemu — — – -

To recap, to install QEMU, here’s what you need to know:

• If you’re okay with having VMs that run ONLY on the architecture same to your host one’s (in my case amd64) and that are accelerated with KVM, run the command: sudo apt install qemu-system-x86 or qemu-system-armorqemu-system-mips, qemu-system-miscor qemu-system-ppcor qemu-system-s390xor qemu-system-sparc(select ONE that corresponds to the architecture of your machine; in my case it is qemu-system-x86)
• If you’re interested in exploring VMs of various architectures use the command: sudo apt install qemu-system, which will install for you a set of emulators that can emulate various architectures supported by QEMU
• If your aim is to create VMs on particular one architecture, QEMU offers several options to choose from, including: qemu-system-arm, qemu-system-mips, qemu-system-misc, qemu-system-ppc, qemu-system-s390x, and qemu-system-sparc.

If you may think that you can just install QEMU with sudo apt install qemuto not bother with the choice with the right package, then sudo apt show qemu will explain why it is not so good idea:

sudo apt show qemu
Package: qemu
##### .... #########
Description: fast processor emulator, dummy package
 QEMU is a fast processor emulator. Once upon a time there was only one
 package named `qemu', with all functionality included. These days, qemu
 become large and has been split into numerous packages. Different packages
 provides entirely different services, and it is very unlikely one will
 need all of them together. So current `qemu' package makes no sense anymore,
 and is becoming a dummy package.
################################

A small note, if you visit the official QEMU project website, you’ll notice they launched version 8.2 last December. But, if you use the apt package manager to install, you’ll end up with QEMU version 6.2. Later I will write a short article on how to build QEMU from its source code, however in this article it is out of scope.

III step: Install user-friendly tools for VMs creation & management

After completing the installation of QEMU in the previous step, you technically have everything necessary to create/run VMs — you have virtualizable CPU ✅, KVM ✅ and QEMU ✅. However, utilizing QEMU as a standalone virtualization tool is for pretty advanced users — having reviewed QEMU’s documentation, it is clear for me that QEMU is far not the easiest option for beginners.

This is where libvirt enters the picture. But what exactly is libvirt?

Libvirt is collection of software that provides a convenient way to manage virtual machines and other virtualization functionality, such as storage and network interface management.

An primary goal of libvirt is to provide a single way to manage multiple different virtualization providers/hypervisors. No need to learn the hypervisor specific tools! Source: Libvirt FAQ

I created this diagram to highlight the logic of core libvirt components:

Libvirt core components and their functionalities (from user perspective)

The libvirt-daemon-system, formerly a part of libvirt-bin, includes an API library and the libvirtd daemon. Similarly, libvirt-clients, another component previously within libvirt-bin, supports three libvirt utilities designed to streamline the management of VMs:

• Virt-manager is a widely used graphical interface tool that allows users to create, use, and manage VMs entirely through a graphical user interface (GUI), facilitating the process of setting VM parameters.
• Virt-install is a command-line tool that enables the creation of straightforward VMs, offering the ability to configure the basic parameters for the VM being set up. Users can then view the VM’s console or desktop environment using virt-viewer.
• Virsh is a command-line utility that stands out for its ability to configure highly advanced VMs. A significant feature of virsh is its capability to manage and modify existing VMs, including the option to connect to them remotely via SSH.

Moving on to the installation of libvirt, it is important to note that installing libvirt without having QEMU previously installed will result in the installation of qemu-system-x86 for you. Proof:

sudo apt install --simulate libvirt-daemon-system
#If qemu-system(-x86) was NOT installed before:
The following additional packages will be installed:
  cpu-checker dmeventd ibverbs-providers ipxe-qemu
  ipxe-qemu-256k-compat-efi-roms jq libaio1 libboost-iostreams1.74.0
  libboost-thread1.74.0 libcacard0 libdaxctl1 libdecor-0-0
  libdecor-0-plugin-1-cairo libdevmapper-event1.02.1 libfdt1 libgfapi0
  libgfrpc0 libgfxdr0 libglusterfs0 libibverbs1 libiscsi7 libjq1
  liblvm2cmd2.03 libndctl6 libnss-mymachines libonig5 libpmem1 libpmemobj1
  librados2 librbd1 librdmacm1 libsdl2-2.0-0 libslirp0 libspice-server1
  libtpms0 liburing2 libusbredirparser1 libvirglrenderer1  
  --->libvirt-clients <---- libvirt-daemon libvirt-daemon-config-network 
  libvirt-daemon-config-nwfilter libvirt-daemon-driver-qemu libvirt-daemon-system-systemd libvirt0
  libxml2-utils libyajl2 lvm2 mdevctl msr-tools ovmf qemu-block-extra
!!-->  qemu-system-common qemu-system-data qemu-system-gui qemu-system-x86
  qemu-utils seabios swtpm swtpm-tools systemd-container
  thin-provisioning-tools
#If qemu-sustem(-x86) was previously installed
The following additional packages will be installed:
  dmeventd jq libdevmapper-event1.02.1 libjq1 liblvm2cmd2.03 libnss-mymachines
  libonig5 libtpms0 ---> libvirt-clients <--- libvirt-daemon
  libvirt-daemon-config-network libvirt-daemon-config-nwfilter
  libvirt-daemon-driver-qemu libvirt-daemon-system-systemd libvirt0
  libxml2-utils libyajl2 lvm2 mdevctl swtpm swtpm-tools systemd-container
  thin-provisioning-tools

To avoid cluttering your OS with unnecessary packages I advise to install the specific version of QEMU you actually nee (e.g qemu-system-x86 or qemu-system-arm) before proceeding with the installation of libvirt.

Additionally, the earlier code snippet demonstrates that installing libvirt-daemon-system will automatically include all essential components of libvirt, including libvirt-clients. Therefore, you only need to execute the following commands to install libvirt, assuming you've already installed the required QEMU:

sudo apt update
sudo apt install libvirt-daemon-system

After installing libvirt-daemon-system, the user that will be used to manage virtual machines needs to be added to the libvirt group. This group is created automatically after installing libvirt but will appear after a system reboot. When you run the command group in the terminal, you should see 'libvirt' listed among the groups (if you check before rebooting post-installation, you might not see it, but it's there).

If the user account you’re using is part of the sudo group, there’s no need to manually add it to the libvirt group, as this is handled automatically. To check if your user account is in the sudo group you can run this command:

getent group sudo
# My output:
sudo:x:27:ubuntu <-- my user 'ubuntu' is in sudo group

However, if the user isn’t a member of the sudo group, you’ll need to add it to the libvirt group manually so it can access system-wide libvirt resources. To achieve this, execute this command :

sudo adduser $USER libvirt

As I mentioned earlier, libvirt operates with its own daemon. To verify if it’s enabled and running, run this command:

sudo systemctl status libvirtd 
# If it is enabled and running, you will see something like this:
(green circle) libvirtd.service - Virtualization daemon
     Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor pres>
     Active: active (running) since Wed 2024-02-07 20:50:21 CET; 10min ago
TriggeredBy: (green circle) libvirtd.socket
             (green circle) libvirtd-ro.socket
             (green circle) libvirtd-admin.socket

If you find it inactive and not running, you’ll need to initiate it:

# First, enable it
sudo systenctl enable libvirtd.service
# Then start it
sudo systemctl start libvirtd.service
# Check if it is running 
sudo systemctl status libvirtd

Going back to libvirt-clients, once you’ve installed libvirt-daemon-system, virsh will be installed automatically. To verify if virsh has successfully connected to the installed QEMU, execute virsh uri. If the output is empty, you'll need to manually connect them using one of the commands provided below:

# My output:
ubuntu@ubuntu:~$ virsh uri
qemu:///system
#If your is empty, then:
#connects locally as the root user to the daemon supervising guest virtual machines on the KVM hypervisor.
virsh connect qemu:///session
#connects locally as a user to the user's set of guest local machines using the KVM hypervisor.
virsh connect qemu:///system

By using virsh, you can check the current versions, the compatibility of virtualization tools, and explore the capabilities of your virtualization setup:

ubuntu@ubuntu:~$ virsh version
Compiled against library: libvirt 8.0.0
Using library: libvirt 8.0.0
Using API: QEMU 8.0.0
Running hypervisor: QEMU 6.2.0
# Another important command, the output is pretty long, you can study it in details
ubuntu@ubuntu:~$ virsh capabilities
<capabilities>
  <host>
    ...
    <cpu>
      <arch>x86_64</arch>
      ...
      <vendor>Intel</vendor>
    </cpu>
    ...
    <iommu support='no'/>
    ...
  </host>
  # First option of emulated architectuire for guest
  <guest>
    <os_type>hvm</os_type>
    <arch name='i686'>
      <wordsize>32</wordsize>
      <emulator>/usr/bin/qemu-system-i386</emulator>
      ...
      <domain type='qemu'/>
      <domain type='kvm'/>
    </arch>
    ...
  </guest>
  # Second option of emulated architectuire for guest (main for me, arch x86_64)
  <guest>
    <os_type>hvm</os_type>
    <arch name='x86_64'>
      <wordsize>64</wordsize>
      <emulator>/usr/bin/qemu-system-x86_64</emulator>
      ...
      <domain type='qemu'/>
      <domain type='kvm'/>
    </arch>
   ...
  </guest>
#If you have installed qemu-system without specific architecture, 
#then you will see here more <guest> options

Create your first VM with virtinst

I mentioned earlier there are two alternative tools to virsh for creating, managing, and running VMs:virt-install along withvirt-viewer and virt-manager. You need to install them if you want to use them:

#this command will install virt-install and virt-viewer
sudo apt get install virtinst 
sudo apt get install virt-manager

I will not use virt-manager since there’re many tutorials available and it’s a tool with a GUI. Instead, I’ll use the virt-install command line tool to create a VM from. i have chosen for test Fedora OS as the operating system of guest VM. You can remove my comments and execute the following commands:

ubuntu@ubuntu:~$ cd Downloads/
#First, I need .iso file of Fedora
wget https://download.fedoraproject.org/pub/fedora/linux/releases/39/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-39-1.5.iso
# I copy Fedora's .iso to the folder, where libvirt stores disks images
sudo cp Fedora-Workstation-Live-x86_64-39-1.5.iso /var/lib/libvirt/images

virt-install \
--name fedora-test \
#RAM
--memory 8192 \
#Number of vCPUs
--vcpus 6 \
# disk size (in GB), this option will automatically create storage disk image
#in qcow2 format - this is an optimized format provided by QEMU
--disk size=20 \
--cdrom /var/lib/libvirt/images/Fedora-Workstation-Live-x86_64-39-1.5.iso \
#virt-install --osinfo list to choose correct existing os-variant, otherwise you will get an error
--os-variant fedora38

Once you hit enter, your virtual machine will be set up, and virt-viewer will automatically start, opening up the desktop environment of the guest operating system. Through virt-viewer, you can then go ahead with installing the chosen OS on your VM.

That’s it! Have fun with your VM, and I look forward to seeing you reading the next parts of this virtualization series! The upcoming part will be especially useful for those who need their VM to utilize the host machine’s GPU (heads up — it doesn’t do this by default).