Fun with Routing Protocols

Use FRRouting and Mininet to learn network routing protocols

You may have heard of BGP, the Border Gateway Protocol, which is fundamental to the working of the Internet by enabling network providers to connect to each other and share information. You might be aware of protocols such as OSPF or ISIS that run on corporate networks. Routing protocols such as these are often considered to be mysterious but critical technology, unavailable to all but a small group of experts. While it is true that network routing protocols can be complex, they are not out of reach for those with a basic understanding of IP networks. What is out of reach is access to a testbed with which to run the routing protocols. How can you really learn something without being able to run experiments? Only reading about protocols omits all of the fun of hands-on learning.

A good solution is to use a network simulator on which we can build virtual networks and run real routing protocols. A simulator enables us to design network backbones with connected subnetworks, configure BGP to share routing information between networks, run experiments, and learn what it takes to get everything to work. This article describes a simple integration of FRRouting and Mininet that makes it easy to run basic examples, and possible to design and configure your own virtual networks.

FRRouting + Mininet

Mininet is a lightweight network emulator that uses Linux containers to simulate networks of routers, switches, and connected hosts. The main focus for Mininet has been working with Software Defined Networking (Openflow, P4), but it works well for more traditional routing technology as well.

FRRouting is an open source routing protocol suite that runs on Linux and Unix platforms and implements key routing protocols such as BGP, OSPF, ISIS, and more. Installing FRR on a Linux system basically gives that system the ability to function as a software based router. Using FRRouting is similar to the experience of being at the console of a Cisco router.

It is relatively easy to use either one of these packages on their own. But there is a little extra work needed to get these to work well together. The best way to use FRR and Mininet together is in a virtual machine custom made for this experience. Using a VM has the benefits of:

• There are no dependencies on your primary OS
• You don’t install FRR or Mininet on your own system
• You can’t mess up your system’s networking, just the networking of the VM
• You don’t run privileged commands (run by root) on your own system

This VM will be based on Ubuntu Server 22.04.4, on which we will build and install FRR, Mininet, and a small set of example Mininet networks and FRR configurations.

Run the Virtual Machine

For those who have not previously done so, working with virtual machines is likely the most challenging task in using the FRR/Mininet integration. The easiest option on Windows is to use Virtual Box, and this short article provides a nice introduction. I use virt-manager over QEMU/KVM running on my Ubuntu desktop system. VmWare has an extensive set of tools for working with VMs.

You can either download the 2.7 GB OVA file to import into Virtual Box or build your own VM. For those using QEMU/KVM, you can convert the OVA file to a QCOW2 file (which can be imported into QEMU/KVM) by following these instructions.

The easiest way to work with FRR/Mininet is by logging into the VM over SSH. If you are using VirtualBox, you will need to add a simple port forwarding rule to enable SSH access. See this article for details.

To build the FRR/Mininet virtual machine, we:

• create a VM
• download and install Ubuntu server
• and download and build FRR and Mininet from source.

Instead of including the detailed instructions here, see this README.md on Github for the steps to create the FRR/Mininet VM. But first, read on to see what you can do once you have the FRR/Mininet integration image.

FRRouting / Mininet Integration

As mentioned before, Mininet uses Linux containers to emulate a network. Mininet creates one container for each emulated router and host. Normally, a single instance of FRRouting runs that takes control of all routing on a Linux device. When using FRRouting with Mininet, we don’t run a global instance of FRRouting, instead we start one instance of FRRouting in the container of each emulated router:

/usr/lib/frr/frrinit.sh start <container name>

We populate a configuration file for each FRRouting instance:

/etc/frr/<container name>/frr.conf

The scripts and configuration files for emulated networks are at https://github.com/jmwanderer/mininet_frr and are included in the VM image in the directory ~/mininet_frr. Each example network has the following contents:

• FRRouting configuration files for each virtual router: r1/frr.con, r2/frr.conf, …
• A script to configure the FRRouting virtual routers: config_frr.sh
• A mininet python script that builds the simulated network and starts the FRR daemons: mn_topo.py

The FRR/Mininet VMs has three example networks:

• A simple 2 router network that uses static routes
• An 4 router network running OSPF
• A wide area network running BGP

To run the examples, login to the VM either by connecting to the VM terminal or using an SSH client. To login, use the username: ubuntu and password: ubuntu, configured during the install of the virtual machine. Change into the directory for a specific example network (e.g. mininet_frr/two_router), and run:

./config_frr.sh
sudo ./mn_topo.py

Two Router Network

This is a simple but non-trivial network with two connected routers, each router connecting a subnet that has a single host.

Two Router Network

The routers are R1 and R2, the hosts D1 and D2. In this network, host D1 uses R1 as the default gateway, and relies on R1 to reach any destination not on the local subnet. Because D1 and R1 are connected on the same subnet, they talk directly to each other. The same is true with D2 and R2, as well as with R1 and R2. However, without additional routing information, R1 does not know how to reach D2, and R2 does not know how to reach D1. D1 and D2 can not communicate.

Static Route Configuration

To make this network function correctly, we configure static routes on R1 and R2. This configuration is stored in r1/frr.conf and r2/frr.conf. If you look at these files, you will find the following critical configuration:

r1/frr.conf:

ip route 10.1.0.0/24 10.100.0.2 r1-eth2

r2/frr.conf:

ip route 10.0.0.0/24 10.100.0.1 r2-eth2

The configuration for R1 says that to reach the 10.1.0.0/24 network (where D2 is located), go through R2 (identified by the IP address 10.100.0.2).

The configuration for R2 says that to reach the 10.0.0.0/24 network (where D1 is located), go through R1 (identified by the IP address 10.100.0.1).

Run the Simulation

These operations need to be done as root (it is good we are using a VM), and you will be prompted for the password at times to grant permission to run as root. Since this is done in a VM, your host system is safe.

We run the simulation with the the following commands:

cd mininet_frr/two_router/
./config_frr.sh
sudo ./mn_topo.py

This starts the mininet emulator and the relevant FRR routing daemons, and brings up the mininet command line. To stop the simulation, type exit or press ctrl-D.

ubuntu@frr-system:~/mininet_frr/two_router$ sudo ./mn_topo.py 
[sudo] password for ubuntu: 
*** Creating network
*** Adding controller
*** Adding hosts:
d1 d2 r1 r2 
*** Adding switches:
s1 s2 
*** Adding links:
(d1, s1) (d2, s2) (r1, r2) (s1, r1) (s2, r2) 
*** Configuring hosts
d1 d2 r1 r2 
 * Starting watchfrr with command: '  /usr/lib/frr/watchfrr -N r1 -d  -F traditional   zebra mgmtd staticd'
 * Started watchfrr
 * Starting watchfrr with command: '  /usr/lib/frr/watchfrr -N r2 -d  -F traditional   zebra mgmtd staticd'
 * Started watchfrr
*** Starting controller
c0 
*** Starting 2 switches
s1 s2 ...
*** Starting CLI:
mininet>

Mininet provides access to the linux shell for each emulated node. For example, test that d1 can reach d2 by sending 3 pings with the mininet CLI:

mininet> d1 ping d2 -c 3
PING 10.1.0.252 (10.1.0.252) 56(84) bytes of data.
64 bytes from 10.1.0.252: icmp_seq=1 ttl=62 time=0.393 ms
64 bytes from 10.1.0.252: icmp_seq=2 ttl=62 time=0.105 ms
64 bytes from 10.1.0.252: icmp_seq=3 ttl=62 time=0.110 ms

To look at the routes configured in the linux routing table on R1:

mininet> r1 ip route
10.0.0.0/24 dev r1-eth1 proto kernel scope link src 10.0.0.1 
10.1.0.0/24 nhid 6 via 10.100.0.2 dev r1-eth2 proto 196 metric 20 
10.100.0.0/24 dev r1-eth2 proto kernel scope link src 10.100.0.1

Test that everything can talk to everything:

mininet> pingall
*** Ping: testing ping reachability
d1 -> d2 r1 r2 
d2 -> d1 r1 r2 
r1 -> d1 d2 r2 
r2 -> d1 d2 r1 
*** Results: 0% dropped (12/12 received)

Run the FRR Console

The FRR console is provided by vtysh, which also needs to be run as root in this configuration. We run vtysh and use the -N option to specify which virtual router to connect. Here we connect to r1:

sudo vtysh -N r1
[sudo] password for ubuntu: 

Hello, this is FRRouting (version 9.1-MyCustomFRR-g45021ae82).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

This is a git build of frr-9.1-50-g45021ae82
Associated branch(es):
        local:stable/9.1
        github/FRRouting/frr.git/stable/9.1

vm-r1#

You can enter commands such as show running-config, or show ip route

vm-r1# show ip route 
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

C>* 10.0.0.0/24 is directly connected, r1-eth1, 00:11:51
S>* 10.1.0.0/24 [1/0] via 10.100.0.2, r1-eth2, weight 1, 00:11:51
C>* 10.100.0.0/24 is directly connected, r1-eth2, 00:11:51

The ? and <tab> characters are useful to see what commands are available.

You can change the configuration of the router with the configure command:

vm-r1# configure  
vm-r1(config)# no ip route 10.1.0.0/24 10.100.0.2 r1-eth2
vm-r1(config)# exit

Here I have removed the route to the 10.1.0.0 subnet, and D1 and D2 can no longer connect. Even if you save this new configuration with the write command (updating: /etc/frr/r1/frr.conf), running ./config_frr.sh will recopy the original configs and restore it to a working state.

Exit or crtl-D to get out of the FRR console. Exit mininet to shutdown the simulation.

OSPF Network

The two router network is very simple, and doesn’t even need FRR to function. Static routes can be configured directly in Linux. But a 4 router 8 subnet configuration is more interesting:

Four router, eight subnet network

Configuring static routes on the 4 routers would work here, but would be hard to maintain. Anytime any subnet anywhere is added or changed, every router would need to be updated. Instead, we run OSPF to share network routing information with all other routers in the network.

This example uses a minimal OSPF configuration that advertises specific attached networks to all neighbors. For example, the OSPF configuration for R4:

router ospf
 ospf router-id 10.7.0.1
 network 10.7.0.0/24 area 0.0.0.0
 network 10.8.0.0/24 area 0.0.0.0
 network 10.102.0.0/24 area 0.0.0.0
 network 10.103.0.0/24 area 0.0.0.0
exit

To run the simulation:

cd mininet_frr/ospf/
./config_frr.sh
sudo ./mn_topo.py

When you run this simulation, you will see that it takes a few moments for the OSPF protocol to run and everything becomes reachable.

BGP Network

Though BGP is a powerful and complex protocol with many details, we can see the fundamentals of BGP working in a simple network of 4 routers connecting 3 separate organizations defined by BGP Autonomous System Numbers.

Three AS Wide Area Network

The BGP configuration is a bit more complex. The configuration specifies the Autonomous System numbers, the specific BGP peers, and specific policies about what information to share and what to accept. As an example: consider the BGP configuration for R1:

router bgp 65008
 bgp router-id 10.1.0.1
 neighbor 10.1.0.2 remote-as 65009
 !
 address-family ipv4 unicast
  network 10.100.0.0/24
  neighbor 10.1.0.2 route-map ALLOW-ALL in
  neighbor 10.1.0.2 route-map ALLOW-ALL out
 exit-address-family
exit

route-map ALLOW-ALL permit 100
exit

To run this simulation:

cd ~/mininet_frr/bgp/
./config_frr.sh
sudo ./mn_topo.py

You will see that BPG takes a few moments to run and share the necessary routing information such that everything is reacha

Customizing the Networks

You can create your own configurations and custom networks. Use vtysh to create or update configurations, and use the write command to save the configuration to frr.conf.

Designing custom networks isn’t easy, but is an excellent way to develop knowledge and experience with networking technology. The steps are outlined in README.md. Pay particular attention to the IP addresses on the subnets and interfaces. It is terrific fun to debug a broken network and find that you typed 10.1.0.1 instead of 10.0.1.1!

To get more information on Mininet, look at the Mininet walkthrough. If you want to go deep into the network protocols, FRR has a User Guide. May your inner network engineer find enlightenment!