Memory hotplug with Qemu/KVM and libvirt

Hotplugging of Memory is supported since version 2.1 of Qemu and is an alternative to memory ballooning. Removing the memory online is supported since Qemu 2.4 and is covered in a different article.

Configuration changes

To tell libvirt that we want to have memory hotplug there has to be a maxMemory element and a NUMA node declaration. If not present, the following lines should be added:

<domain type='kvm'>
  <maxMemory slots='16' unit='KiB'>16777216</maxMemory>
  <cpu>
    <numa>
      <cell id='0' cpus='0-2' memory='1048576' unit='KiB'/>
    </numa>
  </cpu>
</domain>

maxMemory specifies the maximum amount of memory that is allowed to be plugged in and behaves the same way as the memory attribute. The second limitation is the amount of modules that can be plugged. I currently stick to 16 slots as that allows more small increases and smaller steps at unplugging if needed.

libvirt currently enforces a specified NUMA node for memory hotplug. id specifies the node number and cpus which vcpus belong to this node. Even if not all vcpus are active, the maximum amount of vcpus should be specified here. In this case, I have 3 vcpus and libvirt uses a zero-based numbering here. Memory and unit should be as the omitted memory attribute.

Hotplug with virsh

virsh requires an XML file with a memory device. To add 128 Megabyte, the file would look like the following.

<memory model='dimm'>
  <target>
    <size unit='MiB'>128</size>
    <node>0</node>
  </target>
</memory>

To add it to the running guest and also add it to the config the following is needed.

virsh attach-device <vm name> <xml filename> --config --live

Hotplug with python and libvirt API

For Debian based systems python-libvirt or python3-libvirt is needed. On openSuse, the package is named libvirt-python. The following snippet would do everything which the previous virsh command did for us.

import libvirt
conn = libvirt.open()
vm = conn.lookupByName("vm_name")
xml = "<memory model='dimm'><target><size unit='MiB'>128</size><node>0</node></target></memory>"
vm.attachDeviceFlags(xml,
libvirt.VIR_DOMAIN_AFFECT_LIVE|libvirt.VIR_DOMAIN_AFFECT_CONFIG)

Possible problems

ACPI Errors

Section-unaligned hotplug range: start 0x10800000, size 0x640000
acpi PNP0C80:01: add_memory failed
acpi PNP0C80:01: acpi_memory_enable_device() error

Linux requires memory hotplug in sizes of 128Mib or any multiple of it.

No error and nothing happens

There a multiple reasons that nothing happens on guest side

1. kernel too old or modules missing
   For hotplug at least kernel 3.2 is needed and kernel 3.9 to remove it online. Also the modules acpiphp and pci_hotplug are needed.
2. Wrongly configured kernel
   This can be checked wit looking in the config of the running kernel with cat /boot/config-/boot/config-`uname -r` | grep CONFIG_MEMORY_HOTPLUG= which should return a y for this option
3. Memory is still offline
   There a two ways to set the hotplugged memory in an online mode.
   Adding an udev rule:
   SUBSYSTEM==”memory”, ACTION==”add”, TEST==”state”, ATTR{state}==”offline”, ATTR{state}=”online”
   Using a small script:
   for i in `grep -l offline /sys/devices/system/memory/memory*/state`
   do
   echo online > $i
   done