Advanced Docker networking // Custom outgoing IP

A few days ago I noticed that my docker services use my hosts default IP when talking to other internet applications. My host machine has two different outgoing IPs assigned to one interface (eth0, eth0:1) and I wanted my docker services to use the latter when listening for incoming connections as well when connecting to other internet services. Where the first one is a quite common task the latter is a more advanced one.

In this story I’ll show you how to achieve it and I also give you a technical insight into how Docker manages its containers networking.

Have a nice read.

Copyright © 2017 Docker Inc., Source https://github.com/docker/machine

Docker Networking

I didn’t have in depth knowledge about how Docker manages its containers networking. So I went ahead and started reading https://docs.docker.com/engine/userguide/networking/

In the very last paragraph it states that Docker is using iptables for Linux hosts in order to manage its networking. Iptables is a kernel module which enables filtering and modifying network related stuff based on defined rules and commands within so called tables and chains. Each one of these has its own specific purpose and you may already know one of these, e.g. the table “filter” with the chain “INPUT” is commonly used to secure a server by limiting its open ports.

In summary Docker is utilizing the iptables “nat” to resolve packets from and to its containers and “filter” for isolation purposes. Curious how I was I took a glance at the current state of my hosts iptables with the command iptables-save which led to the following output regarding the nat table:

*nat
:PREROUTING ACCEPT [2:139]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER — [0:0]
-A PREROUTING -m addrtype — dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype — dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
COMMIT

We can see that Docker introduced a custom chain named DOCKER and is redirecting specific packets from PREROUTING and OUTPUT to it.

The magic regarding outgoing traffic takes place with the instruction -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE

Masquerade

Let’s break down the magic instruction -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE

This rule

• belongs to the chain -A POSTROUTING which allows modifying routed packets.
• applies to packets coming from the subnet -s 172.17.0.0/16 that are not going to be sent via the interface ! -o docker0 where the latter is Dockers default bridge interface and the former is its IPv4 subnet.
• instructs to jump to -j MASQUERADE which assigns the corresponding IP of the outgoing interface to matching packets.

This rule is crucial in the sense that we want to change that behaviour to have it assign our desired secondary IP instead of automatically assign packets the default outgoing IP.

Let’s do it

To verify that everything is working out I’m using a docker container to query my current IP via http://www.myip.ch .

docker run --rm byrnedo/alpine-curl http://www.myip.ch

The service states that my current IP address is 78.31.xxx.xxx which is my hosts default ethernet IP.

Docker is creating the masquerading rule automatically so we need to disable this creation and handle the iptables part by our own. As I don’t like the idea of changing the behavior of Dockers default bridge docker0 I decided to create a new network named bridge-coi with the following command:

docker network create --attachable --opt ‘com.docker.network.bridge.name=bridge-coi’ --opt ‘com.docker.network.bridge.enable_ip_masquerade=false’ bridge-coi

The option name=bridge-coi specifies the name of the interface and the option enable_ip_masquerade=false disables the automatic creation of such MASQUERADE entries within iptables. (Source)

Okay so we got our network with masquerading disabled therefore we need to handle the iptables part by our own. So let’s add the crucial entry to the iptable nat and its chain POSTROUTING with the following command:

iptables -t nat -A POSTROUTING -s 172.18.0.0/16 ! -o bridge-coi -j SNAT --to-source 5.104.xxx.xxx

Note that -j MASQUERADE is doing an SNAT (source nat) under the hood though automatically choosing the --to-source ip based on the outgoing interface.

This looks a little bit different compared to the previous -j MASQUERADE rule. So let’s break it down again:

This rule

• belongs to the POSTROUTING chain of the nat table, so nothing changed.
• applies to packets coming from the subnet-s 172.18.0.0/16 that are not going to be sent via the interface ! -o bridge-coi where the latter is our new bridge interface and the former is its IPv4 subnet, which you can either specify yourself or check with command docker network inspect bridge-coi .
• instructs to jump to -j SNAT while giving the instruction to assign the source IP 5.104.xxx.xxx to all matching packets.

So let’s try it out and check the configuration by running a container attached to our just created network.

docker run --rm --network bridge-coi byrnedo/alpine-curl http://www.myip.ch

et voilà, the service now states that my current IP address is 5.104.xxx.xxx which is my hosts secondary ethernet IP.

Final words

Despite being an advanced topic we saw that it’s achievable with a little bit of effort. I quite enjoyed working on this and I hope you had fun reading my story.

Due to that this is my first story I highly appreciate any comments and encourage you to criticize me. I’m looking forward to anything you have to say. :)

Joe