Container Networking FAQ for the Developer
I work quite a bit with application developers on the cusp of rethinking how they design applications. Very few of them have experimented with new technologies like containers and often times, they come to me with some interesting questions on (Docker) container networking. In an ideal world, an application developer should not care about how their application connects to other applications or stores some information. In reality, they do need to know a little bit about how their container connects to something and something connects to the container. As a result, I usually get asked the same five questions:
- Why does my container have a 172.17.0.x address?
- Why can’t I access my web server?
- Why can’t my web server connect to my database server?
- How do I know the IP address of my container if it is ephemeral?
- How does software-defined networking (SDN) or an overlay change this?
The intent is to neither explain every detail of container networking nor recommend one method of container networking over another. Instead, this outlines some key concepts for aspiring and experienced developers alike learning how to develop applications for the container space and experiencing problems with connectivity.
Why does my container have a 172.17.0.x address?
By default, Docker will deploy the container to its default bridge network. This bridge, called docker0, is a Linux bridge with a default network of 172.17.0.0/16. Linux bridges are designed to have a local scope on the host, so by default, you will not be able to access it externally. When a container is created, it is assigned to the Linux bridge and therefore, not actually directly available unless you configure it to do so. Docker IP address management will assign the next available address from its network.
Why can’t I access my web server?
The above explanation covers why you will not be able to access your container, it is on an internal Linux bridge. In order to make it accessible, you have to relate the internal bridge to a physical interface. This is done by iptables NAT rules. I will not go into too many details here but conveniently, Docker will configure your iptables rules via the -p option (publish).
All you have to do is tell Docker to publish your ports to the host (host port : container port) and you can access your container via the host’s IP address. In this case, I’m using my localhost to access my web server container on port 80.
There are many other ways to enable Docker to assign a different IP address to your container but each network type has different consequences.
Why can’t my web server connect to my database server?
This is probably the most common question I get. Connecting containers requires more than using a Docker Compose file. Part of the problem is that you have to be conscientious of where your containers are deployed. In a typical application, you can pass an environment configuration that describes the database. If it is a Java application, this is likely a Java option (-D). If this is Python Flask application, it is an environment variable. At runtime, how do you know this is the right hostname and port of your database?
In order for your web server to properly reach your database, you need to check two things:
- The IP address, hostname, or URL that your web server thinks that your database is at.
- The port that your web server thinks that you’ve mapped your database to on the host.
A Docker Compose should look something like this:
web:
build: .
environment:
- DATABASE_HOST=HOST.IP.ADDRESS.XX or BRIDGE.IP.ADDRESS.XX
- DATABASE_PORT=27017
ports:
- "8080:5000"
links:
- db
db:
image: mongo:latest
ports:
- "27017:27017"Using localhost’s typical loopback address (127.0.0.1) will actually resolve to the internal loopback interface of the container. Basically, it’s resolving to the localhost of itself! If you use the loopback address, your web server will not be able to find the database and throw a “connection refused” error, like below.
Use the host’s IP address or expected IP address allocated from the bridge (172.17.0.x). The latter is not the best idea, especially because you may have more containers deployed on the default bridge. The former is not the best idea either because containers might move to another host, which leads us to the next point…
How do I know the IP address of my container if it is supposed to be ephemeral?
In the container space, mapping the actual IP address of containers to the services they represent is a network challenge. The solution is referred to as service discovery. It can be a fairly complex problem to solve. Fortunately, there are a few options in the container orchestration space. Generally, if you choose to use port mapping (like above) and you are testing on your local desktop, the IP address of your container does not change much and it does not need to. It uses the IP address of the host, after all, and it will not change unless your container moves to another host. Most service discovery mechanisms maintain a record of this port mapping, allowing you to resolve to a service name rather than IP address. For example, the services in the Docker Compose above would be recorded in your service discovery mechanism as:
{
"web_app.service": {
"ipAddress": "host.ip.addr.xxxx",
"port": 8080
},
"database.service": {
"ipAddress": "host.ip.addr.xxxx",
"port": 27017
}
}Notice it is not the container’s port that service discovery component records, it is the host port. You would be able to access the database service using database.service, so your web service configurations never needs to know the actual IP address and port of the container.
How does software-defined networking (SDN) or an overlay change this?
The networking paradigm above uses NAT in order to expose the container. While this is helpful to a certain degree, you run the risk of hitting a limit on the number of ports you can map to on your host. Every more confusing, every application must specify a container to host port mapping. If two applications want to use the same ports, the losing application will need map to a new port. This is quite troublesome.
More recently, the networking space has experienced a resurgence in the use of network overlays. A network overlay is a virtual network that sits on an actual physical network. Using a container with an overlay makes deploying the application and service discovery far easier. For example, a web application could be deployed with docker run -d --net=<overlay network name> image without needing to expose the port and without having to worry about how other services access it. The container would have its own IP address in docker inspect, which means your service discovery would use the internal IP address of the container and the container’s port. This sort of configuration is often referred to as “IP-per-container”. Your service discovery component would have a record looking more like this:
{
"web_app.service": {
"ipAddress": "container.ip.addr.xxxx",
"port": 5000
},
"database.service": {
"ipAddress": "container.ip.addr.xxxx",
"port": 27017
}
}You will not run out of ports in this networking paradigm but you may run out of IPv4 address space! Can you test an overlay locally? Maybe. It depends on the tool. You may not even need to test with an overlay if none of the services that interact have conflicting ports. If you have a container orchestration system, overlays do eliminate the need for you to worry about port mapping when you deploy your service to it.
A second benefit to an overlay is the potential for segmentation through software-defined networking. When network engineers refer to segmentation, it means dividing the network in a secure way such that a machine on one network segment cannot reach a machine on a different network segment. In the container space, this becomes more important as a larger number of containers need to speak to each other (and not speak to each other). Imagine these rules:
- My accounting service should be able to talk to my accounts receivable service.
- My accounting service should be able to talk to my accounts payable service.
- Don’t let my accounts payable service talk to my accounts receivable service.
Implementing these rules can be done in a variety of ways (iptables, for example). Software-defined networking (SDN) can facilitate and enforce these rules as needed with its controller. The main advantage of SDN is that it allows one central point of control to organize any of the network flows in the overlay network. SDN can push the following rule set (not exactly depicting the standard but demonstrating the intent):
- Allow IP addresses from source accounting to destination accounts receivable.
- Allow IP addresses from source accounting to destination accounts payable.
- Deny everything else.
SDN allows you to easily map the source and destination IP addresses of your containers and apply flow rules to it. A combination of service discovery and SDN can give you intent-based security rules, making it much easier to control connectivity between containers and the services they represent.
Summary
Software-defined networking or an overlay is certainly not the best answer to every container networking question. Port mapping can fit many different use cases, it all depends on what you are trying to accomplish. Talk to your friendly infrastructure engineer if you run into problems or you have questions on how container networking works. More importantly, use Docker bridge to the best of your ability to test interactions between your services!
Additional References
Container Networking
- Overview of Container Networking
- Docker Bridge — Detailed Reference
- Docker Networking Reference Architecture
- Docker run Reference
- Docker User Guide for Networking
Service Discovery
- New Architecture for Service Discovery
- Service Discovery in Microservices
- Service Discovery & Solutions Overview
