Full Mesh IPIP Tunnels

When observing Calico networking on a new Kubernetes cluster, a mystery presented itself. How do these IP-in-IP tunnels really operate behind the scenes?

Here’s an example of the routing table from a Kubernetes cluster with Calico as the networking plugin:

# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.20.96.1 0.0.0.0 UG 0 0 0 eth0
100.104.161.0 172.20.41.63 255.255.255.192 UG 0 0 0 tunl0
100.109.14.0 0.0.0.0 255.255.255.192 U 0 0 0 *
100.109.14.1 0.0.0.0 255.255.255.255 UH 0 0 0 cali990d7f36226
100.109.14.2 0.0.0.0 255.255.255.255 UH 0 0 0 cali68d727df87f
100.109.14.3 0.0.0.0 255.255.255.255 UH 0 0 0 cali043bfd9b6a5
100.118.59.192 172.20.122.216 255.255.255.192 UG 0 0 0 tunl0
100.120.205.0 172.20.136.173 255.255.255.192 UG 0 0 0 tunl0
100.122.53.192 172.20.74.9 255.255.255.192 UG 0 0 0 tunl0
100.122.235.64 172.20.86.62 255.255.255.192 UG 0 0 0 tunl0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
172.20.96.0 0.0.0.0 255.255.224.0 U 0 0 0 eth0
# ip route
default via 172.20.96.1 dev eth0 
100.104.161.0/26 via 172.20.41.63 dev tunl0 proto bird onlink 
blackhole 100.109.14.0/26 proto bird 
100.109.14.1 dev cali990d7f36226 scope link 
100.109.14.2 dev cali68d727df87f scope link 
100.109.14.3 dev cali043bfd9b6a5 scope link 
100.118.59.192/26 via 172.20.122.216 dev tunl0 proto bird onlink 
100.120.205.0/26 via 172.20.136.173 dev tunl0 proto bird onlink 
100.122.53.192/26 via 172.20.74.9 dev tunl0 proto bird onlink 
100.122.235.64/26 via 172.20.86.62 dev tunl0 proto bird…