Flannel vs Calico : A battle of L2 vs L3 based networking

We discuss today the networking in container world and primarily in context of K8s . We are not covering the policies and isolation part , but only how L2 and L3 play a role in packet flows.

Flannel is an overlay network mechanism where as Calico is basically a pure L3 play.

Flannel works by using a vxlan device in conjunction with a software switch like linux bridge or ovs.

Container A when tries to reach container B on different host the traffic is pushed to the bridge on host A via the VETH pair. The bridge then based on ARP tries to get the mac of container B. Since container B is not on the host the traffic by bridge is forwarded at L2 to the vxlan device (software TAP device) which then allows flannel daemon software to capture those packets and then wrap then into a L3 packet for transport over a physical network using UDP. Also vxlan tagging is added to the packet to isolate them between tenants.

Flannel shown diagrammatically

In case of Calico, the approach is little different. Calico works at Layer 3 and depends on Linux routing for moving the packets.

Calico injects a routing rule inside the container for gateway at this IP 169.254.1.1.