Exposing k8s applications — The components of k8s networking

Still coming to terms with k8s networking, you are not alone? The automation simplicity offered by public cloud providers and workstation versions (minikube & microk8s) hide the network complexity that enables application developers to manage networking functions. Adding in the the redefinition and new terminology around “ingresses”, “load-balancers” and “service mesh”, makes building a private k8s infrastructure that leverages the potential benefits, and is sufficiently robust, a significant undertaking. It’s not possible to have all of the answers because the k8s development moves quickly and the scope is broad, we will focus on a specific problem: how to get traffic in and out of containers for production applications, while maintaining the significant improvements in automation, security and separation of responsibilities that k8s done right can offer.

To get started, let’s cover some important terminology used in k8s networking as it somewhat differs from the typical use of these terms in networking.

1. Service. This is the abstract used to configure how a pod is exposed as a network service. Unless a service is configured, the pod is not accessible outside of the complex.
2. Ingress. The definition is an API object that manages external access to services in a cluster. Most of the discussion is around ingress controllers and there are plenty of them. At its most basic level an ingress controller in traditional lexicon is an Application Load Balancer or a proxy server deployed in a pod. These terms are used interchangeably throughout k8s. We will try not to add to the confusion.
3. Load Balancers. In k8s a load balancer is an external device that directs traffic to nodes, either directly to application pods or ingress controllers. In the future these will be referred to as Gateways so we will use that term.
4. Container Network Interface (CNI). As k8s is an orchestration system providing abstraction to the underlying hardware infrastructure, the CNI is responsible for the orchestration of the networking between pods and nodes.

These components work together to get traffic to, from and between PODs, and while some may suggest that the terminology is confusing, it is not inaccurate. The confusion comes from the overuse of the term “Load Balancer”.

In reality there is lots of load balancing happening using different techniques throughout a k8s system and external to k8s but required for operation. Some like to align this load balancing with layers in the network stack but this can also be confusing because the techniques are applied and reapplied as traffic transits the complex. Some simple explanations of load balancing components can help.

• DNS load balancing. Multiple entries are put in the dns for a single FQDN and requests get different responses. Its use is common and is the foundation of scalability techniques such as Global Anycast, however it has its pros and cons. Most specifically the ability to rapidly update DNS caches can result in unreachable addresses. ks8 recommends the use of DNS within the complex for service delivery, and applications need to add DNS entries to be exposed externally to external dns servers, so these mechanisms play an important part in developing an overall solution.
• Network Load Balancing. This function is undertaken by the switches and routers used to construct the network. In the context of load balancing this is occurring at the IP layers of the network. There are of course other lower layer load balancing techniques however these are transparent to the k8s infrastructure. Network load balancing is a straightforward process where the routing table has a number of equal cost entries to a destination and balances traffic amongst those destinations either equally or weighted.
• Ingress Controllers. Generally speaking these are higher level TCP/HTTP load balancers. While many have other functions at their basic level they operate either on a TCP flow basis, just like network load balancing or interact with HTTP to distribute connections. At this level ingress controllers can provide other interesting functionality like higher level traffic routing based upon URLs and an array of management and redundancy solutions. As they operate a session level, their operation is different from network load balancing because they manage connections.

These are the visible load balancing components. Under the covers a kube-proxy (a k8s component), also acts as a load balancer providing connectivity to the control plane, internal connectivity between pods and also to by default it will distribute traffic to the correct node/pod combination when traffic arrives at any node. When you combine all of these things and also the potential that tunneling has been used to establish connectivity between PODS(the default for most CNIs,) you can start to appreciate the network complexity.

There will be at least two networks in a k8s system, possibly more. K8s has two networks, the serviceSubnet providing access to services within the cluster, and PodSubnet provisioned by the CNI and provides cluster networking. If tunneling is being used by the CNI, connectivity between nodes is handled by a network of tunnels autoconfigured between nodes. Without tunnel connectivity between nodes and to the POD subnet is the responsibility of host networking.

There are a lot of choices in CNI’s. If you’re building with private infrastructure, CNI is an important decision, however they all have a similar mix of functionality, connectivity and security/policy. Putting aside the very network-specific CNI’s (eg. MPLS), the primary connectivity decision is the use of tunnels or integrate with the network (usually referred to as flat networking). It is a subject of a longer discussion and there isn’t a right or best answer. Tunneling is the default on the popular CNI’s because it requires no additional integration with the underlying network; automatically created tunnels make it transparent. A flat network design requires integration with routing demanding design and planning. A flat network design forces network complexity to be addressed at the start of the deployment. Using tunnels avoids the complexity at the start, however as the system scales, tunneling increases complexity in operation and failure modes. I am a fan of flat networking, however it does require significant skill to implement. The second component of CNI’s is policy which is another way of saying firewall rules, but that is selling the functionality short. A network policy system in kubernetes doesn’t look like a firewall, the policies are defined by services/applications using labels and selectors from the application configuration. This is a significant improvement over the traditional creation and management of rules, and can contribute to a significantly more secure infrastructure as the automation makes securing workloads easy and avoids errors.

Kubeproxy. A conversation on CNI and networking is not complete without a discussion on kubeproxy, a key component of k8s that provides connectivity for the control plane and configures the distribution of traffic between nodes and containers. The default and initial implementation of kubeproxy configures iptables, uniformly on each node. A quick look at a k8s node will help illustrate the scope of these iptables rules. Every service creates iptables rules so there is a practical limit to the number of services that can be hosted in a single complex, as well as the impact to performance from large scale iptables use. In an effort to address this in a uniform way, there is an upstream (k8s source) alternative for kubeproxy based upon IPVS that improves performance and scale. According to some, this is not enough and there are kubeproxy implementations (both open source and proprietary) that add additional functionality as well as performance. However a divergence from the upstream kubeproxy’s should be considered carefully as it is a core component of the k8s platform and divergence will dictate other architectural choices now and in the future.

Finally, a few words on Service Meshes. Put simply, a service mesh is a k8s internal network proxies integrated with the PODs that run applications. Network traffic enters the service mesh via ingress controllers running the proxy and the other key component is a proxy also running in each pod. As you already know, a pod can consist of multiple containers, so these proxies (referred to as “sidecars”) are also installed in the POD. The POD is configured to pass all traffic in and out via the sidecar proxy. This forms the Service Mesh. While it sounds like it adds another layer of complexity (and it does) there are some useful benefits. Passing traffic through proxies increases flexibility in how traffic is distributed among application pods making version control, blue/green and redundancy modes possible. Proxies also instrument application visibility avoiding the need for application-specific instrumentation. Proxies also usually deal with encryption and as all traffic transits an ingress proxy and the sidecar proxy, traffic can be encrypted in flight inside the system. All of this requires software that orchestrates these actions such as Istio, as previously mentioned it adds more complexity to the system, however its worth weighing the tradeoff.