Kubernetes Network Plugins

6 min readJul 1, 2019

What are CNI Plugins?

CNI (Container Network Interface) is a project by CNCF that defines a specification which allows communication between containers. Kubernetes supports CNI plugins for the communication between pods.

In the CNI architecture, orchestrator (Kubernetes) can ADD or DEL CNI plugins using CNI libraries.

Kubernetes comes with kubenet for the communication by default. It is typically used within cloud providers. However, this plugin is very basic and does not implement advanced features that comes with CNI plugins. Also, kubeadm does not support kubenet. Therefore, you should use a CNI plugin.

Kubernetes imposes following rules for network communication:

All containers can communicate with all other containers without NAT
All nodes can communicate with all containers (and vice-versa) without NAT
The IP that a container sees itself as is the same IP that others see it as

As long as implementations satisfy the rules, the architecture can differ. Therefore, there are a lot of different plugins with different architectural approaches. We will examine Flannel, Calico, Weave, Cilium, and kube-router.

Terminology

For a better understanding between different plugins, it may be helpful to learn underlying architectures of these plugins.

kube-proxy

Kube-proxy and kubelet runs in all the worker nodes. Kube-proxy allows clients to connect to the defined services.

Kube-proxy has three mods:

Userspace

This mod was used in the first implementations of kube-proxy. Kube-proxy was named with proxy thanks to this mod. It adds rules to iptables so that all communication redirects through proxy server. It is no longer used since it is much slower compared to the other modes.

Iptables

This mode adds rules to iptables so that iptables redirects straight to pods without using a proxy server. It is the default mode of kube-proxy.

IPVS

IPVS (IP Virtual Server) is layer-4 load balancer inside the Linux kernel. It is built on top of netfilter like iptables. It utilizes hash table instead of chain as in iptables. Therefore, it scales much better than iptables.

Network policy

It is a Kubernetes specification that can be used to control traffic between pods. It requires a CNI plugin that supports network policies to work. If there is no CNI plugin or plugin does not support, policies will not have an effect on the system. It is very useful for increasing security in Kubernetes.

Figure 6: Example: DENY all traffic from other namespaces

Overlay Networks

An overlay network abstracts a physical (underlay) network to create a virtual network. It provides simpler interface by hiding complexities of the underlay.

VXLAN

VXLAN is a network tunneling protocol in the Linux kernel. Network tunneling means hiding protocol (VXLAN) within another protocol (TCP/IP). VXLAN tunnels layer 2 frames inside of layer 4 UDP datagrams. This creates an illusion that containers on the same VXLAN are on the same L2 network.

Linux network structures in VXLAN:

veth: Virtual ethernet pair, it connects network namespaces

bridge: It is used to connect ethernet pairs in Linux

vtep: VXLAN tunnel endpoint, it’s the entry/exit point for VXLAN tunnels

BGP — Bird

AS (Autonomous system): A network or networks regulate by an organization

BGP (Border Gateway Protocol): The routing protocol of the internet. It is used for routing between autonomous systems. This usage is called eBGP (exterior BGP). If it is used within an AS, it is called iBGP (interior BGP).

BGP based overlay networks uses iBGP.

To use BGP as overlay network, hosts in the cluster are set as BGP peers. Each peer needs to have a daemon which allows sharing routing information between peers.

BIRD (BIRD Internet Routing Daemon) is a daemon which supports BGP as well as other protocols. Calico uses this daemon to implement BGP based overlay network.

BGP does not have extra tunneling cost and this may help to achieve better network performance. However, if you do not have the control of the underlay network (e.g. public cloud), you need to use tunneling (IP-IP encapsulation or VXLAN).

BPF — XDP

BPF (Berkeley Packet Filter) was initially conceived in 1992 as to provide a way to filter packets and to avoid useless packet copies from kernel to userspace. It was used in small number of applications such as tcpdump. Then, in 2013, there was a new version developed called eBPF (extended BPF) which adds new functionalities and improves performance of the original BPF. This allowed BPF programs to be developed for much wider use cases. Therefore, BPF generally refers to eBPF nowadays while cBPF (classic BPF) refers to the old version.

XDP (eXpress Data Path) is a data path recently added to Linux kernel. It relies on eBPF to perform fast packet processing.

CNI plugins generally use kube-proxy or directly iptables for routing. However, Cilium is based on BPF and XDP to provide a faster and more scalable option.

Plugins

Flannel

one of the most popular plugins
provides VXLAN tunneling solution
configuration and management are very simple
does not support Network Policies
has a mode called host-gw which provides a tunnelless solution as long as hosts are connected with direct layer 2 connectivity

Calico

one of the most popular plugins
default choice of the most of Kubernetes platforms (kubespray, docker enterprise, etc.)
uses BGP and Bird, a daemon called Felix configures routes on Bird
supports IP-IP encapsulation if BGP cannot be used
supports Network Policies
uses iptables for routing but it can be configured to use kube-proxy’s IPVS mode
has a CLI tool named calicoctl

Weave

provides VXLAN tunneling solution
all of the nodes are connected as mesh which allows it to run on partially connected networks
does not scale well because of the mesh structure
stores configuration files on pods instead of Kubernetes CRDs or etcd
has an encryption library
supports Network Policies
has a CLI tool called weave

Cilium

provides VXLAN tunneling solution but it can be used with kube-router to provide a tunnelless solution
uses BPF and XDP for routing
supports Network Policies
also has Cilium Network Policies which provides functionality not yet supported in Kubernetes Network Policies (e.g. HTTP request filters)
has a CLI tool called cilium
Linux kernel must be at least 4.9

Kube-router

uses BGP
uses IPVS for routing
simpler and smaller than Calico (one daemonset vs Felix)
supports Network Policies

Conclusion

For most cases, Calico should be enough. It is widely supported, has Network Policies, uses BGP which is faster than VXLAN, and so on. Cilium can be used if security is important. If you want a quick and easy solution, you can go with Flannel. In the end, you should choose the plugin according to the needs of your project.