Open vSwitch and Libvirt

This post was originally posted on blog.aaronorosen.com on Feb 21, 2012.

Recently I started playing with libvirt in order to manage all of my virtual machines since it’s able to handle most hypervisors through one API. One issue I ran into though is that I use Open vSwitch as my bridging mechanism. For those of you who do not know what Open Vswitch is, it’s basically a virtual switch that has a ton of cool features like OpenFlow, 802.1Q , sFlow, and many other things. I use Open vSwitch mostly as a software OpenFlow switch for my thesis work. Open vSwitch has something called brcompat which makes brctl compatibly with OVS bridges. This said it seems like libvirt is not compatible with it. For example when I used virsh in order to start a vm and attach it to an OVS bridge it said, “error: Failed to add tap interface to bridge ‘br0’: Invalid argument”. Though, when I ran brctl by hand I could attach the tap interface to br0 just fine. I’m not sure if this is a bug or I was missing something in my setup.

Anyways.. Libvirt now has support for interfacing with OVS bridges, so I was interested in trying that out. The first step to doing this was to get the latest libvirt source code with these new changes. Since the underlying host runs gentoo linux and provides a live-ebuild for libvirt I started using that. Unfortunately the ebuild for that is not up to date with the current source in libvirt repo. In order to fix this I needed to add an overlay which required the following steps:

mkdir -p /usr/local/portage/profiles/
echo "my_local_overlay" > /usr/local/portage/profiles/repo_name
Added PORTDIR_OVERLAY="/usr/local/portage/" to  /etc/make.conf
mkdir -p /usr/local/portage/app-emulation/libvirt
cp /usr/portage/app-emulation/libvirt/libvirt-9999.ebuild \
    /usr/local/portage/app-emulation/libvirt/
cp -r /usr/portage/app-emulation/libvirt/files \
   /usr/local/portage/app-emulation/libvirt/

Now libvirt-9999.ebuild needed to be corrected. Thankfully someone already figured out what needed to be changed and filed a bug report here (also thanks to hasufell from #gentoo to pointing me to this!) so I was able to easily make those changes and finally compile libvirt successfully.

ebuild libvirt-9999.ebuild manifest
emerge app-emulation/libvirt

(I also emerged the virt-manager from ebuild live though I don’t think the newest version is required. )

of course I could have just cloned libvirt directly from source:

git://libvirt.org/libvirt.git

though I wanted portage to do all of the heavy lifting for me such as handling dependencies, etc. This also lets me use smart-live-rebuild in order to pull the latest source code for all of my live-ebuilds so I don’t need to manage them all individuality.

Now to actually using Open vSwitch and libvirt. Open vSwitch can be downloaded from:

git clone git://openvswitch.org/openvswitch

after getting the source follow INSTALL.Linux for instruction on how to build and configure it. First, I created a br0 and attach eth0 to it.

ovs-vsctl add-br br0
ovs-vsctl add-port br0 eth0

Next, I changed the interface section of the configuration file for the vm telling it to use my OVS bridge interface.

virsh edit <domain>

<interface type='bridge'>
    <mac address='52:54:00:43:1f:f4'/>
    <source bridge='br0'/>
    <virtualport type='openvswitch'>
    </virtualport>
</interface>

Now, when the vm is started it will be added to the OVS bridge. As I was saying before I use Open vSwitch as an OpenFlow switch to experiment with, though I also run several other virtual machines for different tasks on the network (Cacti/Nagios VM, etc). In order to experiment with things on the network and keep other VMs unaffected I create a special ovsbr datapath and attach a patch port between ovsbr and br0.

ovs-vsctl add-br ovsbr
ovs-vsctl add-port br0 patch-to-ovsbr
ovs-vsctl set Interface patch-to-ovsbr type=patch
ovs-vsctl set Interface patch-to-ovsbr options:peer=patch-to-br0
ovs-vsctl add-port ovsbr patch-to-br0
ovs-vsctl set Interface patch-to-br0 type=patch
ovs-vsctl set Interface patch-to-br0 options:peer=patch-to-ovsbr

After doing that I also enable OpenFlow on the ovsbr datapath and point that at my OpenFlow controller:

ovs-vsctl set-controller dp0 tcp:130.127.39.177:6633

This allows me to try out crazy things on the ovsbr and isolate this from the rest of the VMs attached to br0.