Advanced Docker networking // Custom outgoing IP
A few days ago I noticed that my docker services use my hosts default IP when talking to other internet applications. My host machine has two different outgoing IPs assigned to one interface (eth0, eth0:1) and I wanted my docker services to use the latter when listening for incoming connections as well when connecting to other internet services. Where the first one is a quite common task the latter is a more advanced one.
In this story I’ll show you how to achieve it and I also give you a technical insight into how Docker manages its containers networking.
Have a nice read.
Docker Networking
I didn’t have in depth knowledge about how Docker manages its containers networking. So I went ahead and started reading https://docs.docker.com/engine/userguide/networking/
In the very last paragraph it states that Docker is using iptables for Linux hosts in order to manage its networking. Iptables is a kernel module which enables filtering and modifying network related stuff based on defined rules and commands within so called tables and chains. Each one of these has its own specific purpose and you may already know one of these, e.g. the table “filter” with the chain “INPUT” is commonly used to secure a server by limiting its open ports.
In summary Docker is utilizing the iptables “nat” to resolve packets from and to its containers and “filter” for isolation purposes. Curious how I was I took a glance at the current state of my hosts iptables with the command iptables-save
which led to the following output regarding the nat
table:
*nat
:PREROUTING ACCEPT [2:139]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER — [0:0]
-A PREROUTING -m addrtype — dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype — dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
COMMIT
We can see that Docker introduced a custom chain named DOCKER and is redirecting specific packets from PREROUTING and OUTPUT to it.
The magic regarding outgoing traffic takes place with the instruction -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
Masquerade
Let’s break down the magic instruction -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
This rule
- belongs to the chain
-A POSTROUTING
which allows modifying routed packets. - applies to packets coming from the subnet
-s 172.17.0.0/16
that are not going to be sent via the interface! -o docker0
where the latter is Dockers default bridge interface and the former is its IPv4 subnet. - instructs to jump to
-j MASQUERADE
which assigns the corresponding IP of the outgoing interface to matching packets.
This rule is crucial in the sense that we want to change that behaviour to have it assign our desired secondary IP instead of automatically assign packets the default outgoing IP.
Let’s do it
To verify that everything is working out I’m using a docker container to query my current IP via http://www.myip.ch .
docker run --rm byrnedo/alpine-curl http://www.myip.ch
The service states that my current IP address is 78.31.xxx.xxx which is my hosts default ethernet IP.
Docker is creating the masquerading rule automatically so we need to disable this creation and handle the iptables part by our own. As I don’t like the idea of changing the behavior of Dockers default bridge docker0
I decided to create a new network named bridge-coi
with the following command:
docker network create --attachable --opt ‘com.docker.network.bridge.name=bridge-coi’ --opt ‘com.docker.network.bridge.enable_ip_masquerade=false’ bridge-coi
The option name=bridge-coi
specifies the name of the interface and the option enable_ip_masquerade=false
disables the automatic creation of such MASQUERADE
entries within iptables. (Source)
Okay so we got our network with masquerading disabled therefore we need to handle the iptables part by our own. So let’s add the crucial entry to the iptable nat
and its chain POSTROUTING
with the following command:
iptables -t nat -A POSTROUTING -s 172.18.0.0/16 ! -o bridge-coi -j SNAT --to-source 5.104.xxx.xxx
Note that -j MASQUERADE
is doing an SNAT (source nat) under the hood though automatically choosing the --to-source
ip based on the outgoing interface.
This looks a little bit different compared to the previous -j MASQUERADE
rule. So let’s break it down again:
This rule
- belongs to the
POSTROUTING
chain of thenat
table, so nothing changed. - applies to packets coming from the subnet
-s 172.18.0.0/16
that are not going to be sent via the interface! -o bridge-coi
where the latter is our new bridge interface and the former is its IPv4 subnet, which you can either specify yourself or check with commanddocker network inspect bridge-coi
. - instructs to jump to
-j SNAT
while giving the instruction to assign the source IP5.104.xxx.xxx
to all matching packets.
So let’s try it out and check the configuration by running a container attached to our just created network.
docker run --rm --network bridge-coi byrnedo/alpine-curl http://www.myip.ch
et voilà, the service now states that my current IP address is 5.104.xxx.xxx which is my hosts secondary ethernet IP.
Final words
Despite being an advanced topic we saw that it’s achievable with a little bit of effort. I quite enjoyed working on this and I hope you had fun reading my story.
Due to that this is my first story I highly appreciate any comments and encourage you to criticize me. I’m looking forward to anything you have to say. :)
Joe