An Architect’s view on Network Virtualization

This is a two-part post — Part 1 (this post) introduces the components associated with MidoNet network virtualization architecture and discusses the concept of moving complexity to network edges. Part 2 describes distributed databases, overlay model and traffic flow through the virtual network.


There has been a dramatic shift in networking — landscapes have changed and workloads are no longer stationary on physical servers. Dynamic workloads are now in containers or Virtual Machines, moving between inter and intra site locations. The introduction of virtualization removed the certainty of server and corresponding workload locations. As a result, network designs must adapt to the new style of workload fluidity. This type of elasticity puts pressure on traditional networks as it changes where the state and intelligence are positioned. Network state and rate of change are key elements affecting network performance and efficiency.

To support new workload requirements, the network must become elastic and have the ability to dynamically follow workloads around the infrastructure. To accomplish this, a network overlay model exists consisting of a physical underlay and logical overlay. The physical underlay is the traditional physical network — its only concern is with IP endpoint reachability and doesn’t get involved with endpoint policies. It may take the form of a traditional Core, Aggregation and Access or a Leaf and Spine. The overlay is the software part and rides on top of the physical infrastructure. The overlay is software based and combined with a distributed architecture, maps the intelligence of the network.

The introduction of the overlay meets the elasticity requirements and simplifies network management. The complexity of the network now moves to the software layer at the edge of the network, allowing the physical underlay to simply forward packets as fast as possible. Pushing the complexity to the edge of the network allows the network to be managed in a more efficient way. It eliminates the box by box mentality and utilizes a centralised view of the network.

Introducing MidoNet

Midokura support this type of architecture with their MidoNet solution. MidoNet is an open source network virtualization overlay that decouples the Infrastructure as a Service (IaaS) from the physical network. They offer the overlay software component and not the physical underlay. It’s completely distributed and replaces the OVS plugin with a MidoNet plugin taking over all the networking functions in OpenStack. MidoNet plugs into Neutron as a monolithic plugin and maps Neutron models to MidoNet constructs.

The diagram below displays the generic MidoNet components.

The solution contains two main elements. The first component is the Network State Database (NSDB), consisting of two open source databases — Zookeeper and Cassandra. Both of these databases have different functions and roles. Zookeeper holds the topology information (virtual network and host mapping) while Cassandra stores things like NAT entries and connection tracking for backup purposes. The second component, the MidoNet Agent which runs in userspace, is installed on the actual hypervisor compute and gateway nodes. As it runs in userspace, it communicates through Netlink with the Linux kernel and uses the OVS kernel module to implement data plane forwarding. The agent also runs on the gateway nodes for external connectivity. The agent on the gateway and compute nodes are exactly equivalent: they both run the MidoNet agent and the OVS kernel module.

The gateways are physical hosts which own ports at the edge of the MidoNet virtual topology and leverage Quagga. The only difference is that for compute nodes, these ports usually connect to VMs, whereas in gateways these ports are mapped to physical NICs that connect the cloud to other networks in addition to the cloud datapath. They are typically configured with Border Gateway Protocol (BGP) to connect to different administrative domains. BGP is the de facto routing protocol used to control route propagation between internal and external networks.

These components together create a software-based network abstraction layer between hosts and the physical network. It enables the move from hardware-based appliances to network functions in software based in a multitenant distributed virtual domain. MidoNet is a distributed network virtualization solution that transforms the physical network.

The following sections detail some of the key difference between MidoNet architecture and traditional networks.

  • Complexity to the Edge
  • Distributed Databases
  • Overlay Model
  • Traffic Flow

Complexity to the Edge

Midokura overlay architecture pushes the complexity to the edge of the network, leaving the underlay to concentrate on forwarding packets as quickly as possible. The architectural concepts employed by Midokura are not new. Request for Comments: 3439 by R. Bush and D. Meyer outlines some guidelines to which architectures should adhere to. In short, it demonstrates that complexity of the network belong to the edges and the only way for networks to scale is to keep complexity out of the core.

Service Provider MPLS networks further prove this architectural success and changed the way data planes and control plane interrelated. The “intelligence” was moved to the edge with the MPLS Provider Edge (PE) routers who carry out best path decision. The Provider (P) routers sit in the core of the network and focus on swapping labels. There is still control plane activity in the core but a lot of the complexity is moved to the PE routers. The internal P routers do not have client end to end visibility, all they require is reachability to ingress and egress PE nodes for the Label Switch Path (LSP). The ingress and egress PE nodes carry out all the path decisions.

MPLS networks shifted the way we think about control planes. By pushing the control plane responsibility to the edges, while it didn’t eliminate the challenges of a distributed control plane, it’s certainly reduced the complexities. With the simple core and complex edge design, networks are able to scale to an Internet size level.

The MidoNet agent is located at the edge of the network. All the complexity handling is carried out via distributed MidoNet agents. It is installed on the physical hosts and helps connect the real world physical ports to virtual ports. As mentioned, the MidoNet agent is the same on the compute host as the gateway host. But at setup, you can use the templates to set up the MidoNet agent for the host of gateway node by using the MidoNet agent resource template.

Run the following command to configure the MidoNet resource template:

$ mn-conf template-set -h local -t TEMPLATE_NAME
Replace TEMPLATE_NAME with one of the following templates: agent-compute-large agent-compute-medium agent-gateway-large agent-gateway-medium

In a virtual world you could say that the edge of the network is now the boundary between the physical and virtual ports. All traffic gets processed at the edges, where it ingresses the physical network. The virtual devices become distributed and packets can traverse a particular virtual device at any host in the cloud. The architecture enables distributed virtual bridges, routers, NATs, FWs, LBs and DHCP servers. The capacity of your network grows with the addition of more hosts — each host will process the network packets that appear on the ports it owns. Capacity management is simple and you don’t need to add processing power to an individual node. This gives you endless scalability with zero single points of failure.

In the physical world, data traffic is processed by the physical Ethernet ports on physical machines. In a virtual world consisting of virtual machines, the traffic is handled by virtual ethernet ports. In most cases, traffic from virtual ethernet ports needs to be sent to the physical network to reach external destinations. To support this type of connectivity, there are software constructs creating a virtual tie between physical and virtual world ports.

Just like in the physical world, the virtual switch has ports and interfaces to send and receive traffic. In a Linux environment, we have both tap (logical) interfaces and physical interfaces.

In a MidoNet world, we have MidoNet virtual ports, which is a port that exists in the MidoNet virtual topology and stored in MidoNet’s Zookeeper database. It belongs, therefore, to a MidoNet virtual device, such as a virtual router or a virtual bridge. Any Linux network interface, indistinctively of being a tap or a physical NIC, can be mapped to a MidoNet port. This is a known as a port binding.

Once configured, the MidoNet agent on a gateway host learns about the binding and issues a call to the local datapath to add the interface as a netdev port. When the new datapath port is added to the datapath and if it’s operational state is up, the MidoNet agent will publish to ZooKeeper that the virtual router port is up and located on hostX.

The screenshot below shows midonet-cli output of binding listings for two edge gateway hosts.

While moving policies closer to the core reduces the number of devices that enforce policies, it also reduces the effectiveness of policy steering throughout the network. Policy dispersion at the edge of the network is a lot more effective than policy dispersion at the core.

About Matt Conran

Matt Conran is an independent consultant, blogger, and the publisher of He is a 16 year veteran in the networking industry with core skillsets in Data Centre, WAN, MPLS, security and virtualization technologies. He has implemented multiple infrastructure projects with startups, governments, enterprises, and service provider customers, including being a lead Network Architect in major global greenfield deployments. He loves to travel and has a passion for landscape photography.

Originally published at on January 26, 2016.

Show your support

Clapping shows how much you appreciated Midokura’s story.