Kubernetes Networking With Calico

Kubernetes Network Introduction

Kubernetes is a network paradigm that helps to deliver consistency and simplicity across a variety of networking environments and implementations. The Kubernetes network model lays the groundwork for understanding how Kubernetes containers, pods, and services connect with one another. Each pod has got an IP address. Containers within it share that ip address and communicate. Pods can communicate with each other using the Kubernetes Pod network.

Network Isolation is done by network policy. Network policies are Kubernetes resources that control the traffic between pods and/or network endpoints.

Kubelet is a Node agent running in each node. Kubernetes has limited built-in network support called kubenet. It provides some basic network connectivity, but it’s much more common to use third-party Network Implementations which plug into Kubernetes using the CNI API (Container Network Interface).CNI config files are used to determine which CNI plugins to run. There are different kinds of CNI plugins and they can be chained together. For example, network connectivity versus IP address management or IPAM. Calico provides both network connectivity and IP management plugins.The way this works is, when Kubernetes is creating a new pod, it calls the Calico network plugin which in turn calls the Calico IPAM plugin. The IPAM plugin allocates an IP address for the pod and returns it to the network plugin. The network plugin then sets up the pod’s networking with the IP address, and connects it to the pod network, before returning the IP address to Kubernetes, so it can be included on the pod resource. Calico also can provide network support and network policy in different CNI platforms by GKE, Amazon, and Azure CNI.

Kubernetes Services

Kubernetes Services are a network service that abstracts access to a set of pods. A label selector is typically used to define the collection of parts that support each service. When a client attempts to connect to a Kubernetes service, the connection is intercepted and load balanced to one of the service’s Pods.

There are three types of Kubernetes services. They are NodePort, ClusterIP and LoadBalancer. Cluster IP services are the usual way of accessing services from inside the cluster. A cluster IP is a virtual IP address used to represent the service. The way this works is the pod can find the cluster IP using DNS. The pod then tries to connect to the cluster IP, kube proxy will intercept the connection, and load balance it to a backing pod. Kube proxy is responsible for communication of pods in different nodes. Kubeproxy is responsible for managing all the ip tables. LoadBalancer services use a load balancer as a layer in front of Node Ports, to provide a more sophisticated way to access a service from outside the cluster.

Network Policy

Network policy is the primary tool for securing a Kubernetes network. It allows us to easily restrict the network traffic in our cluster, so only the traffic we want to flow is allowed. To understand the significance of network policy, let’s briefly explore how network security was typically achieved prior to network policy. In the past , security was provided by designing a physical topology of network devices such as switches, routers, and firewalls.

Kubernetes defines a standard network policy API, so there’s a base set of features we can expect on any cluster. But it doesn’t do anything with the network policy other than store it. It’s the network plugins that are responsible for actually enforcing the policy.

Let’s look at why network policy is important.

• Attacks continue to grow in sophistication and volume.
• Traditional firewalls struggle with the dynamic nature of Kubernetes.
• Network Policy is label selector based -> inherently dynamic.
• Empowers teams to adopt “shift left” security practices.

Network policy can be thought of as a firewall implemented into the pod network that protects each pod. Calico can take it a step further by enforcing network policies on the nodes themselves.This is equivalent to installing a firewall on each of the node’s network interfaces.

These APIs are referred to as host endpoints by Calico. They, like pods, can be tagged, and network policy is applied to them based on their labels. Again, it’s similar to pods. It’s really important to define egress and ingress rules for a pod. Actually defining them secures the Kubernetes network from the attackers. In K8s, there are two policies. Network Policy and Global Network Policy. Global network policies are valid for all the pods in all the namespaces. And network policy is for certain specific namespaces.

POD Connectivity

The pods also have IP addresses. Each pod has its own networking environment isolated from the host using Linux network namespaces.

The pods are connected to the host using a pair of virtual ethernet interfaces often referred to as a verth pair. The pod see eth0 is their interface, and the host has a generated interface name for each pod beginning with “cali”. Calico sets up the host networking namespaces to act as a virtual router. The local pods are connected to the virtual router, and Calico makes sure the virtual router knows where all the pods are across the rest of the cluster, so it can forward traffic to the right places. All the traffic forwarding happens natively within the Linux kernel through code paths that have been highly optimized over many years. So, the simplicity of this setup pays off in terms of performance.

Traffic between pods on the same node is routed locally and traffic between pods on different nodes is routed over the underlying network. In most networks, it’s as simple as that — the packet that leaves the pod is the packet that goes on the wire. But there are some environments where the underlying network does not know how to forward the pod traffic. In these cases, we need to run an overlay network. Calico supports both VXLAN and IPIP overlays, implemented as virtual interfaces within the linux kernel. When a pod sends a packet to a pod on a different node, the original package is encapsulated using VXLAN or IPIP into an outer packet using the node IP addresses, and hiding the pod IPs of the original inner packet. The underlying network then just handles this like any other node to node traffic. On the receiving node, the VXLAN or IPIP packet is de-encapsulated to reveal the original packet, which is delivered to the destination pod. This is all done in the Linux kernel, so it’s about as efficient as it can be. It still represents an overhead which you might want to avoid if you’re running particularly, network intensive workloads. With this in mind, Calico also supports cross-subnet overlay modes, where nodes in the same subnet can send pod traffic to each other without using an overlay. The overlay is only used when the pod traffic needs to flow between nodes in different subnets, so you get the best possible performance for each network flow only using overlay for the packets that actually need it.

Data Encryption With Calico

We can think of WireGuard as another kind of overlay network option, but with the added benefit of encryption. Calico uses a virtual interface for WireGuard traffic. This encrypts on the sending node, sends the encrypted data over the network to the other node, where the WireGuard virtual interface decrypts the packet, so it can be forwarded on to the destination pod. Calico automates all of the configuration, and provisioning of WireGuard for us. So we can literally turn it on, and off, with a single configuration setting.

IP Pool & BGP In Calico

IP Pools are Calico resources which define ranges of addresses that the Calico IP address management and IPAM CNI plugin can use. The range of addresses is specified using CIDR notation. To improve performance and scalability, Calico’s IP address management allocates IPs to nodes in blocks. Blocks are allocated dynamically to nodes as required. If a node uses up all the Ips from within the blocks it’s been allocated, then additional blocks will be automatically allocated to it. Depending on how we configured our cluster, if there’s no remaining unallocated IP blocks, then a node can borrow individual unused IP addresses from another node’s blocks. In addition to IP address management, IP pools are used to specify IP address range specific networking behaviors. For example, whether to use an overlay mode for pods allocated IP addresses from the pool, or whether to NAT outgoing connections from the cluster, mapping pod IP addresses to their corresponding node Ip address, which is required whenever our pod IPs are not routable outside of the pod network. For example, if the pod network is an overlay network. We can also mark a pool as disabled if we don’t want it to be used for IP address management. Finally, we can specify a node selector that limits which nodes the IP addresses management can be used on, which allows us to limit specific nodes to using specific ranges of IP addresses for their pods.

BGP is a standard-based network protocol. It’s supported by most network routers, and used to power much of the internet. It’s used to share and synchronize routing information between routers. Using BGP, Calico can share these routes with other BGP capable routers in the underlying network. Each node is able to share its local routes to the underlying network, so pods become first-class citizens on the network, and the need for overlay networks is avoided.

Kube-Proxy In Kubernetes

K-proxy is responsible for communication of pods in different nodes. Kubeproxy manages all the ip tables for enabling the communication between the nodes. Alternatively, we can use Calico’s eBPF data plane, which has built-in native service handling so we don’t need to run kube-proxy at all. kube-proxy is a network proxy that runs on each node in the cluster, implementing part of the Kubernetes Service concept. kube-proxy maintains network rules on nodes. These network rules allow network communication to the Pods from network sessions inside or outside of the cluster.

Summary

In Kubernetes, there is limited network support called kubenet. So, we use a third party for networking in Kubernetes. Calico is one example of such a third party. Calico plugs into the Kubernetes using the CNI API. Calico provides lots of network support such as network policies, data encryption methods, pod connectivity, etc.

Every pod has got an IP address. The containers within the pod share the ip address and communicate with each other as the local host, because all the containers within the pod get the same IP address and a different port number. Pods can communicate with each other using the Kubernetes network. In communication between nodes, kube-proxy is useful. It handles all the IP address tables and so on. Calico provides a data plane with ebpf which can be used instead of the kube proxy.

Network isolation can be done using network policies. Kubernetes has its own network policy and Calico has some added features there. Network policies are very important for security. So, when defining the network policy, it’s important to add ingress and egress rules. For data encryption, calico provides a virtual interface called WireGuard.

Calico provides a big support to enhance the kubernetes network. We have discussed some of the main features in this blog post.