Simulating enterprise networks in development using the Docker networking stack
Universe in a Bottle
Have you ever watched Men in Black? Do you remember the cat (its name is Orion, in case you are wondering) with the universe in a glass marble on it’s collar? Well, I’m here to tell you that you, yes you, can have the same thing. Just swap out “universe” for “enterprise network”, and “glass marble” for “development machine”. Not as crazy as MIB, but hey, I’m a pragmatist.
https://www.youtube.com/watch?v=P7ojSW5pODk
Here at Tenable we need to work with many different machines, all chatting away to each other. We need to be able to work with anywhere from a lot of traffic to very little, since there are many different use-cases for the products we build. We are a network security company, so we need to work with networks big and small.
At the same time, just like any other software engineering organization, we need to have predictable development environments which allow us to maintain flexibility, while not changing state out from under our feet. In order to be effective as software engineers, we need to have a development environment where you can have inputs and data, from which you can set your watch on, so that you can have expected outputs that meet customer requirements. Environments that have multivariate changes are ones that are hard to create software with. Yes, sometimes you want more of a hostile environment to harden your work (see Chaos Monkey), but you want to still be in control. Use unit testing as an example. You typically want clear inputs and assertions, but you can also employ other testing like fuzzing, etc. for hardening.
Reaching both flexibility and predictability is hard to do when you have a bunch of different machines all talking to each other. Flexibility and predictability diverge as scale increases when creating development environments like these.
Seeing what the customer sees
As I had said in the introduction, we need to work with a lot of different machines, network protocols, topologies, etc. When you squint at it at a macro level, networks share the same characteristics; you just have a bunch of machines connected to one another. However, we all know that each network is unique. Each network has a different structure, and this is especially true when we are discussing the differences between strictly IT and IT/OT networks.
It is important that we are able to understand use cases, empathize with the customer, and align ourselves as close as possible with different customer environments and topologies when we are creating software. At Tenable that means being able to reproduce different scenarios that our customers have while we develop our products. Creating an environment to work on Tenable products is very much about reproducing different networking scenarios.
Environment Trade-offs
As you can imagine, just like anything else in tech and life, there are trade-offs to strategies for how to set up a development environment which is focused on creating network monitoring software. When looking at any project or task, two opposing forces are always a variant of a greater theme of “cost vs. quality”. This theme can be expressed in many different ways, but in this case we will use “effort vs. fidelity”.
While cost/effort can be measured quantitatively, quality/fidelity are much more qualitative; and all attributes can mean different things to different people. Here we are measuring effort in the amount of configuration and time required for setting up a given solution. Fidelity is determined by how close a given solution matches a real-world scenario; especially the network characteristics of a system.
Let’s take a look at how this theme would play out over 3 different options: a real replica of a physical network, a set of VMs, or by using Docker.
Replicated Physical Network
Creating a physical network for individual development purposes is pretty impractical; the monetary investment alone makes it infeasible. This is also a pretty rigid solution to make changes to, in order to test with different topologies, etc. It does provide the most accurate representation of a true customer network though.
Virtual Machines
In this case, I am referring to a hypervisor on a LAN that engineers have access to. This solution is much more convenient than a physical network, but still requires using resources that are likely shared amongst a team. It still takes a bit of configuration and setup in order to make changes to a virtual network. The interfaces provided by a hypervisor are still pretty robust, and there is also more dedicated compute power than on a laptop, etc.
Containers (Docker)
This solution can run directly on an engineer’s machine, whether in a slim hypervisor for Mac or Windows or directly on the Linux kernel. If Docker is running solely on an engineer’s machine, resources are much more limited than the other options above, and therefore the fidelity isn’t as high as the other solutions. Where this option makes up for the lower fidelity, is that the level of effort to make changes to a simulated network is very low. These changes can be localized to an individual’s machine, so that it doesn’t affect any other team members. Highly configurable, but not as close to a production environment.
I want to emphasize that I am only using a single engineer’s workstation to compare to the other 2 options. Docker has a bunch of network plugins and configuration options to get it to perform like the other options above, and obviously is high fidelity when using something like Kubernetes in a production environment.
Another option that we have not discussed above is the usage of cloud providers such as AWS, Google Cloud, or Azure. Utilizing a VPC with one of these providers also allows for similar flexibility of a software-defined network much like Docker Compose. You may want to take a further look into a solution like this for development environments where you’d need to have a larger footprint.
These 3 solutions we covered offer different levels of either closely matching a production environment, or speed of configuration and flexibility to change. This is where Docker Compose shines for replicating networking scenarios. We can simulate the environment we’d like, localized on our own machine so that we don’t affect other people, and we can make changes very quickly. The environment becomes commoditized; it can be destroyed and rebuilt within a minute or so.
How does Docker Compose match up with other container solutions?
Docker Compose has much less complexity than Kubernetes or other orchestration tools, in that it does not handle scaling in the way some of the other tools do. Ultimately, Docker Compose just gives you a YAML file which acts as a manifest to stitch services together as a holistic system. In the case that I am proposing here, that system isn’t a 3-tier application stack, but is a mini-enterprise environment.
One of the great things about Docker Compose, is that you can quickly repurpose a Dockerfile or image to be used in different ways. For instance, you can run multiple Nessus Network Monitor applications all running in different subnets, and provide PCAP transmitters for each subnet. You can have a network sandbox running on your laptop. You can swap out local directories to mount and spin up new containers, all very easily.
Networking in Docker
In order to mimic an enterprise network, you’ll need to either produce traffic manually or grab ahold of some PCAPs to play them to the containers that you’ve spun up. You don’t want to need to spin up a bunch of Docker containers in order to get some traffic to observe; that defeats much of the purpose for this sort of lightweight development environment. The setup that I described in the last section is one that I run. As you can see in the diagram below, I have 3 networks: public, OT, and IT.
Both OT and IT have PCAP players which localize playing traffic to containers within those subnets. The network I describe above can be translated to a docker-compose.yml file; which is the manifest to connect and run the separate containers.
---
version: '3'
services:
#-----------------
# PUBLIC
#-----------------
ind_sec:
image: industrial-security
ports:
- "18837:8837"
networks:
public:
ipv4_address: 10.111.220.11
#-----------------
# IT
#-----------------
nnm_IT:
image: nnm
depends_on:
- ind_sec
ports:
- "18835:8835"
networks:
public:
ipv4_address: 10.111.220.21
IT:
ipv4_address: 10.111.221.21
nessus:
image: nessus
ports:
- "18834:8834"
networks:
public:
ipv4_address: 10.111.220.23
IT:
ipv4_address: 10.111.221.23
IT_player:
build: ./docker/pcap_player
depends_on:
- nnm_IT
environment:
- loop=4
- speed=2
volumes:
- ./docker/pcap_player/pcaps/IT:/tmp/pcaps
networks:
IT:
ipv4_address: 10.111.221.31
#-----------------
# OT
#-----------------
nnm_OT:
image: nnm
depends_on:
- ind_sec
ports:
- "18836:8835"
networks:
public:
ipv4_address: 10.111.220.22
OT:
ipv4_address: 10.111.222.22
OT_player:
build: ./docker/pcap_player
depends_on:
- nnm_OT
environment:
- loop=4
- speed=2
volumes:
- ./docker/pcap_player/pcaps/OT:/tmp/pcaps
networks:
OT:
ipv4_address: 10.111.222.31
#-----------------
# TAP
#-----------------
tap:
build: ./docker/pcap_player
volumes:
- ./docker/pcap_player/pcaps:/tmp/pcaps
networks:
public:
ipv4_address: 10.111.220.32
IT:
ipv4_address: 10.111.221.32
OT:
ipv4_address: 10.111.222.32
command: bash
networks:
public:
driver: "bridge"
ipam:
config:
- subnet: 10.111.220.1/24
IT:
driver: "bridge"
ipam:
config:
- subnet: 10.111.221.1/24
OT:
driver: "bridge"
ipam:
config:
- subnet: 10.111.222.1/24The OT_player and IT_player are the transmitters for each of the 2 respective subnets. Once instantiated, they will run tcpreplay and broadcast traffic to the NNM container within their subnet. You’ll also notice that we are repurposing the same pcap_player Dockerfile for a “tap” container. This container spans all subnets, and along with the useful packages installed via the pcap_player, it can act as a point to observe or direct traffic all from one place.
The Dockerfile for the PCAP transmitter (pcap_player) is actually pretty straightforward.
FROM centos:6ENV LOOP 1
ENV SPEED 1
RUN yum -y install initscripts
RUN yum -y install epel-release
RUN yum -y install tcpreplay
RUN yum -y install tcpdump
RUN yum -y install iftop
RUN yum -y install bridge-utils
RUN yum -y install nmap
RUN mkdir -p /tmp/pcapsWORKDIR /tmpCMD tcpreplay -i eth0 — mbps $SPEED — loop $LOOP /tmp/pcaps/*.pcap
It installs tcpreplay as the core tool to use, but also some other utilities that are nice to have to observe and provide details. The `/tmp/pcaps` directory is created to mount PCAPs with the `volumes` property in the YAML file.
Bridging the gap
In the example that we just went through, you’ll notice that all of the networks are using the “bridge” driver. This driver is a virtual network that allows containers in the defined network to communicate with each other, but also provides separation from the host machine. Something to keep in mind: because the “bridge” driver is solely a software implementation of lower-level TCP/IP, you may need to configure your bridge network to handle the packets that you are transmitting correctly.
This means that you may need to adjust settings, such as the MTU for the network or the speed at which your PCAP transmitters are running in order to prevent dropping packets. The “tap” container is a good place to run traceroute, or test that the expected traffic is being transmitted by using tcpdump and inspecting the results.
This is a key point to where a certain amount of fidelity is lost when simulating network traffic with this solution. However, make sure you take a look at some of the other drivers and driver plugins that are available; there is a lot of flexibility for different use cases.
Conclusion
Matching a development environment to a production environment is never a perfect science, nor should it be. A development environment has different intended needs that must be addressed differently than, say, if you were sitting on some manufacturing plant floor, connected to the network, and running unit tests and hitting breakpoints. What software engineers can and should have, is the ability to pull and push the levers of effort vs. fidelity. Setting up networks within the Docker stack gives engineers just another set of levers and options.
… and if you are using Docker containers and would like to secure them, make sure to take a look at Tenable.io Container Security.
