Behind the Talk, Part 1: BDD, Docker Networks, Ansible, & Open vSwitch

I had a blast speaking at All Things Open — I won’t go through the entire talk here, instead I will link to the slides — feel free to ask me questions! In summary, to start thinking and treating your network like you would Docker containers, you need to start testing it.

Realization #0: Technologies, protocols, standards, and buzzwords will change. Testing is a way to understand what you expect of your network and whether or not those technologies fulfill those expectations.

I wanted to create a sample to demonstrate just how you can use unit tests, service/contract tests, and integration tests to evaluate your network. I remember as a wide-eyed, fresh-faced engineer that testing was daunting and I had no idea how to get started. Examples from more experienced developers saved me from some poor testing practices. In the spirit of paying it forward, I wanted to create an example of my own. However, it ended up being more difficult than I thought.

What you see at the talk, complete with the fist pump.
Behind the talk, I did not fist pump.

I decided to describe a bit of what goes on behind the talk — specifically the sample code I created for it. As rewarding as it was, it was not without its struggles.

How I Started

First, I wrote my test. Well actually, it was a slide in my deck and I drew from it for my code sample.

Given I have a gateway to my datacenter
When I create a container or software-defined network
Then I should set up a route from that network to my datacenter

That looks an awful lot like Behavior-Driven Development test…which it is, at the end of the day. My initial thoughts when I started to build the example above:

  1. Set up a switch image using Vagrant.
  2. Configure a VLAN on the switch.
  3. Connect two VMs to it.
  4. Write tests to make sure the VMs can reach each other. (AKA smoke testing)

However, the reality looked more like:

Phase 1: Set up switch image with Vagrant. Phase 2: ?. Phase 3: Test.

As it turned out, I did not have a switch image to use, since I no longer had access to my vendor switch images. The irony of this was not lost on me.

Realization #1: I speak on the importance of practicing automation on systems, yet how can someone hone their skills when there isn’t easy access to tools? What should we do as a community to educate with accessibility in mind?

More of a reason to try to use something open source — while it would not fully depict a vendor offering, I realized that it was about accessibility and to a certain degree, extensibility.

Create Switch-Like Image.

At one point, I thought that creating a single mininet switch in a VM would suffice. Not the case. mininet allows you to create an entire network for mocking but it does not fully mimic the types of configuration you might make to a live switch. It was developed to be immutable, for the most part, meaning the configuration the mininet network is booted with tends to be a hardened configuration you test and destroy. As a result, it did not lend well to kind of end-to-end configuration management testing I wanted to demonstrate.

I resorted to repurposing something I already had, namely the Vagrant Open vSwitch (OVS) image I created for some Docker testing (from my very first blog post!).

Realization #2: I might not have had the foresight 10 months ago but the fact that I can reuse it now is a statement to how version control, documentation, and immutable images can save you time if you think about them from the beginning.

Now, what to do with Open vSwitch? I knew I had to set up a container or software-defined network somehow…but how? What did I already know about container networking and SDN?

  1. Container networks tend to use bridges, unless they are set up with SDN.
  2. SDN can be tedious and complicated to set up on a laptop.
  3. Container networks, at least for Docker, need some sort of network driver to register and deregister workloads.

I could handle the first but the last two were blockers. I didn’t want to set up SDN or write my own driver to register workloads — that was just way too much overhead. Plus, I only have two weeks to do it. After some searching, I came across a neat lab from Kubernetes describing how to do a primitive form of multi-host networking. No SDN, uses a bridge network, and doesn’t require a Docker network driver. Plus, it outlines the OVS commands I need to use. My new idea:

  1. Set up two Vagrant boxes with my Open vSwitch image.
  2. Configure a GRE tunnel between them.
  3. Connect a container to each host.
  4. Write tests to make sure the containers can reach each other.

Seems straightforward enough!

Not So Much.

The automation portion had some issues. Namely:

  1. I needed to install the Linux bridge package on my Vagrant box. Easy enough to add the package during the install with a yum install bridge-utils.
  2. The Linux bridge package does not have an Ansible module. Easy enough to use the commands module.
  3. I had to customize the veth interfaces, bridge names, and more to simplify the automation. To ensure uniqueness, I ended up using time stamps. I also figured out that you can name your Docker network’s bridge with the option, so I just used the Docker network’s name. It made it so much easier for me to pass the container network’s name as an extra variable to the Open vSwitch bridge name.
  4. I had to find out a way for Ansible to reach the hosts. Borrowing from my experience with Chef, I just copied the output of vagrant ssh-config and put it in my local SSH configuration. No fuss, no configuring the hosts file.
  5. It takes a long time to install everything and boot the box. So painful to wait 10 minutes. I made it easier on myself and just packaged it as a local Check out the openvswitch_box step in the makefile. Could I have used an openvswitch container? Sure, except it has been about two years since it has been updated and creating a container within a container just seemed too much to me.

Then, I needed to write my tests.

I wanted to know if my Ansible playbook even worked. It was fine to run it manually but I like making lots of changes to my configuration. Plus, I got really tired of typing vagrant up each time I wanted a clean box to check my logic.

  1. I need some kind of client or interface to automatically bring up my testing machines. I did not want to write my own wrapper around Vagrant command line, so I found a nifty library called python-vagrant that covered what I needed. I usually hesitate on libraries that have not been updated in more than a year but I was willing to see if worked for my purposes. I did write a wrapper in case I wanted to switch out the library and I put that under my lib/ folder.
  2. I needed a way to trigger the test. This is where my unit tests needed a testing framework. I like using pytest for unit testing Python. I called my vagrant wrapper so I could create a clean, new box and check to see if the Ansible playbook ran to completion. That code is located under tests/unit.
  3. While the unit test only needed one host, I need two hosts to run a smoke test that checks connectivity between two containers. Fortunately, Vagrant supports multi-machine so I can declare two hosts in one Vagrantfile. I put this Vagrantfile in my test fixtures for my smoke test, since that is the only time I am going to use a multi-machine configuration.
  4. I needed to trigger my playbook when a Docker network is created. Uh-oh. Oh no. I’m not writing my own network driver for this one…What am I going to do now? Even worse, I have to write some contract tests to make sure that the trigger works…
At this point, I got stuck.

Basically, in order to even demonstrate my smoke test, I needed to automate the creation of a Docker network that could trigger my playbook. This problem required my existing knowledge of Docker networks — it was a pretty fun puzzle to solve. I was even able to reuse the content of my previous blogs. Stay tuned for Part 2 about how I figured out my event trigger. In the meantime, check out the ovs-vagrant configuration that installs Open vSwitch on a box and the Ansible playbook which automates the K8s lab.

The References

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.