AWS Firecracker, Guest Networking Using libvirt

3 min readMar 6, 2019

To begin some theory. We don’t need to use libvirt to make Firecracker work. We can, like in a prior tutorial, simply make a tap interface and perform a NAT configuration via iptables. However I find libvirt to be helpful and common in production KVM environments around the world; so I consider it reliable and secure enough for my needs. Its inclusion in our stack does change the threat model slightly and create issues like our DHCP server being exposed to possibly malicious guests but I consider it an acceptable risk for my personal use. Also each Firecracker VM needs its own tap interface but can share a common bridge. This also means it’s a great option for mixing Firecracker and QEMU+Libvirt guests like in my case i wish to use virt-manager to interact with QEMU reverse engineering VMs while debugging samples inside Firecracker for extra isolation.

Before starting make sure you install libvirt and get virsh functional. On fedora 29 you would do:

$ sudo dnf install bridge-utils libvirt virt-install

An example of my network configuration used with a custom microvm:

# Dump Libvirt config with virsh
$ virsh net-dumpxml Firecracker-Test > Firecracker-Test.xml
$ cat Firecracker-Test.xml
<network>
  <name>Firecracker-Test</name>
  <uuid>7f70f7fa-4051-11e9-b210-d663bd873d93</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr1' stp='on' delay='0'/>
  <mac address='EE:CC:00:00:AA:BB'/>
  <domain name='Firecracker-Test'/>
  <ip address='192.168.233.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.233.128' end='192.168.233.254'/>
    </dhcp>
  </ip>
</network>

In the above XML “virbr1” is a bridge interface controlled by libvirt. It will handle DHCP and the NAT configuration so we can be lazy… I mean efficient!

Configuring libvirt is done with “virsh”. After saving the above xml or creating your own, import it.

#import xml
$ virsh net-define Firecracker-Test.xml#Set bridge to start at boot, optional
$ virsh net-autostart Firecracker-Test#start the birdge for first time
$ virsh net-start Firecracker-Test

Now you should have a working “virbr1” interface that starts on boot. In order to use this interface we need to create a tap and associate it with the bridge just like libvirt does when a QEMU+KVM guest boots. This also means any QEMU+KVM guests associated with the “Firecracker-Test” bridge will be able to talk to our Firecracker vm and vice versa.

I use the following bash scrip to configure a tap and join it to a bridge. This step needs to be repeated for each interface needed by a VM.

tap='ftap0' 
br='virbr1'#Change whoami if needed, should be the user firecracker runs as
sudo ip tuntap add dev $tap mode tap user $(whoami)#Add the TAP interface to the bridge:
sudo ip link set $tap master $br#bring up interfaces
sudo ip link set dev $br up
sudo ip link set dev $tap up

If the above is successful you should see something like this in brctl:

#list bridges
$ brctl show
bridge  name bridge id    STP  enabled interfaces
virbr1  8000.525400e70e9d yes  ftap0
                               virbr6-nic

Now edit your boot method for the Firecracker microvm to include the newly created tap interface:

curl --unix-socket /tmp/firecracker.socket \
-X PUT 'http://localhost/network-interfaces/eth0' \
-H 'accept:application/json' \
-H 'Content-Type:application/json' \
-d '{
"iface_id": "eth0",
"guest_mac": "AA:FC:00:00:00:AB",
"host_dev_name": "ftap0"
}'

Lastly if you have a regular ext4 filesystem image, the easiest way to configure the guests networking without booting is to mount the image and edit the network directly.

#Validate the image type we are looking for
$ file rootfs.ext4
rootfs.ext4: Linux rev 1.0 ext4 filesystem data, UUID=8aaa2424-3a0c-4de6-96b1-710369189b5b (extents) (64bit) (large files) (huge files)$ mount rootfs.ext4 /mnt
$ cd /mnt/etc/network/
$ nano interfaces

Static IP options will give you a faster boot but dhcp makes for good client/test only systems. Alpine has a good tutorial on configuring its network.

Well thats all there really is to it! libvirt makes life extremely easy when it comes to getting up and running. It’s a dependency I don’t mind adding in my setup thanks to the enormous usability benefits.

AWS Firecracker, Guest Networking Using libvirt

Written by The Pawlrus