Behind the Talk, Part 1: BDD, Docker Networks, Ansible, & Open vSwitch
I had a blast speaking at All Things Open — I won’t go through the entire talk here, instead I will link to the slides — feel free to ask me questions! In summary, to start thinking and treating your network like you would Docker containers, you need to start testing it.
Realization #0: Technologies, protocols, standards, and buzzwords will change. Testing is a way to understand what you expect of your network and whether or not those technologies fulfill those expectations.
I wanted to create a sample to demonstrate just how you can use unit tests, service/contract tests, and integration tests to evaluate your network. I remember as a wide-eyed, fresh-faced engineer that testing was daunting and I had no idea how to get started. Examples from more experienced developers saved me from some poor testing practices. In the spirit of paying it forward, I wanted to create an example of my own. However, it ended up being more difficult than I thought.
I decided to describe a bit of what goes on behind the talk — specifically the sample code I created for it. As rewarding as it was, it was not without its struggles.
How I Started
First, I wrote my test. Well actually, it was a slide in my deck and I drew from it for my code sample.
Given I have a gateway to my datacenter
When I create a container or software-defined network
Then I should set up a route from that network to my datacenter
That looks an awful lot like Behavior-Driven Development test…which it is, at the end of the day. My initial thoughts when I started to build the example above:
- Set up a switch image using Vagrant.
- Configure a VLAN on the switch.
- Connect two VMs to it.
- Write tests to make sure the VMs can reach each other. (AKA smoke testing)
However, the reality looked more like:
As it turned out, I did not have a switch image to use, since I no longer had access to my vendor switch images. The irony of this was not lost on me.
Realization #1: I speak on the importance of practicing automation on systems, yet how can someone hone their skills when there isn’t easy access to tools? What should we do as a community to educate with accessibility in mind?
More of a reason to try to use something open source — while it would not fully depict a vendor offering, I realized that it was about accessibility and to a certain degree, extensibility.
Create Switch-Like Image.
At one point, I thought that creating a single mininet switch in a VM would suffice. Not the case. mininet allows you to create an entire network for mocking but it does not fully mimic the types of configuration you might make to a live switch. It was developed to be immutable, for the most part, meaning the configuration the mininet network is booted with tends to be a hardened configuration you test and destroy. As a result, it did not lend well to kind of end-to-end configuration management testing I wanted to demonstrate.
I resorted to repurposing something I already had, namely the Vagrant Open vSwitch (OVS) image I created for some Docker testing (from my very first blog post!).
Realization #2: I might not have had the foresight 10 months ago but the fact that I can reuse it now is a statement to how version control, documentation, and immutable images can save you time if you think about them from the beginning.
Now, what to do with Open vSwitch? I knew I had to set up a container or software-defined network somehow…but how? What did I already know about container networking and SDN?
- Container networks tend to use bridges, unless they are set up with SDN.
- SDN can be tedious and complicated to set up on a laptop.
- Container networks, at least for Docker, need some sort of network driver to register and deregister workloads.
I could handle the first but the last two were blockers. I didn’t want to set up SDN or write my own driver to register workloads — that was just way too much overhead. Plus, I only have two weeks to do it. After some searching, I came across a neat lab from Kubernetes describing how to do a primitive form of multi-host networking. No SDN, uses a bridge network, and doesn’t require a Docker network driver. Plus, it outlines the OVS commands I need to use. My new idea:
- Set up two Vagrant boxes with my Open vSwitch image.
- Configure a GRE tunnel between them.
- Connect a container to each host.
- Write tests to make sure the containers can reach each other.
Seems straightforward enough!
Not So Much.
The automation portion had some issues. Namely:
- I needed to install the Linux bridge package on my Vagrant box. Easy enough to add the package during the install with a
yum install bridge-utils.
- The Linux bridge package does not have an Ansible module. Easy enough to use the commands module.
- I had to customize the veth interfaces, bridge names, and more to simplify the automation. To ensure uniqueness, I ended up using time stamps. I also figured out that you can name your Docker network’s bridge with the
com.docker.network.bridge.nameoption, so I just used the Docker network’s name. It made it so much easier for me to pass the container network’s name as an extra variable to the Open vSwitch bridge name.
- I had to find out a way for Ansible to reach the hosts. Borrowing from my experience with Chef, I just copied the output of
vagrant ssh-configand put it in my local SSH configuration. No fuss, no configuring the hosts file.
- It takes a long time to install everything and boot the box. So painful to wait 10 minutes. I made it easier on myself and just packaged it as a local
openvswitch.box. Check out the openvswitch_box step in the makefile. Could I have used an openvswitch container? Sure, except it has been about two years since it has been updated and creating a container within a container just seemed too much to me.
Then, I needed to write my tests.
I wanted to know if my Ansible playbook even worked. It was fine to run it manually but I like making lots of changes to my configuration. Plus, I got really tired of typing
vagrant up each time I wanted a clean box to check my logic.
- I need some kind of client or interface to automatically bring up my testing machines. I did not want to write my own wrapper around Vagrant command line, so I found a nifty library called python-vagrant that covered what I needed. I usually hesitate on libraries that have not been updated in more than a year but I was willing to see if worked for my purposes. I did write a wrapper in case I wanted to switch out the library and I put that under my
- I needed a way to trigger the test. This is where my unit tests needed a testing framework. I like using pytest for unit testing Python. I called my
vagrantwrapper so I could create a clean, new box and check to see if the Ansible playbook ran to completion. That code is located under
- While the unit test only needed one host, I need two hosts to run a smoke test that checks connectivity between two containers. Fortunately, Vagrant supports multi-machine so I can declare two hosts in one Vagrantfile. I put this Vagrantfile in my test fixtures for my smoke test, since that is the only time I am going to use a multi-machine configuration.
- I needed to trigger my playbook when a Docker network is created. Uh-oh. Oh no. I’m not writing my own network driver for this one…What am I going to do now? Even worse, I have to write some contract tests to make sure that the trigger works…
At this point, I got stuck.
Basically, in order to even demonstrate my smoke test, I needed to automate the creation of a Docker network that could trigger my playbook. This problem required my existing knowledge of Docker networks — it was a pretty fun puzzle to solve. I was even able to reuse the content of my previous blogs. Stay tuned for Part 2 about how I figured out my event trigger. In the meantime, check out the ovs-vagrant configuration that installs Open vSwitch on a box and the Ansible playbook which automates the K8s lab.