Managing Ingress Traffic on Kubernetes Platforms

Why you need an Ingress and how to pick the right one

Photo by Brendan Church on Unsplash

On Kubernetes platforms, traffic may come from an internal source (between workloads within the cluster), or an external source (from an outside client to a workload in the cluster). The latter, often referred to as north-south traffic, is a major source of security risk. It is critical to manage ingress traffic.

For this undertaking, generations of tools have emerged. Here, I’ll take a look at the problems that brought about these tools, available options, and their limitations. As a note, this article is concerned with ingress traffic only, and assumes the traffic arriving at the edge of cluster has passed basic security examinations, such as Web Application Firewall (WAF) and network firewall. Plus, in our context concerning traffic management, the term ingress (as well as Ingress controller and ingress rules) should not be confused with “ingress” as a policy type in the context of NetworkPolicy.

There are currently four families of tools in the Kubernetes ecosystem to manage ingress traffic into the cluster:

• Kubernetes Service (with LoadBalancer type)
• Kubernetes Ingress
• Ingress CRD
• Gateway API

I list them in chronological order. The earliest and simplest is Kubernetes Service object. It simply allows external traffic to cross over into the cluster, without really managing the traffic.

Kubernetes Service

Kubernetes Service resource has four ServiceTypes (ClusterIP, NodePort, LoadBalancer and ExternalName). Only the LoadBalancer type is a realistic way to expose workload outside of the cluster. In contrast, ClusterIP and ExternalName types function within the cluster. NodePort exposes workload by opening a point of entry at each node, which is not recommended beyond sandbox environment.

A Service resource with LoadBalancer type directs traffic from external client to internal Pods. To achieve this, several activities happen under the hood. First, the Cloud Controller Manager provisions a load balancer appliance (at a separate cost) from the cloud provider. At the frond end, this load balancer secures an IP address routable from clients. The back end targets the node port on each Kubernetes node. Then, at the node, kube-proxy listens to the node port and from there forwards incoming traffic to destination pod running on the node itself. The Service controller automatically synchronizes the configurations between the load balancer and the node ports. This type of Service resource operates at the transport layer, agnostic of application-layer messages. It opens a “gate” at the cluster border, but doesn’t check what content, be it harmless or malicious, is carried into the cluster. In production, we rarely use this option by itself.

For security, we need more than always-open gates. We need to inspect and control the untrustworthy traffic across the gate into the cluster. For example, we often want to terminate TLS to examine where web request attempts to go. Within the cluster, we may also want to route the request only to its permitted destination. In some cases, we might event want to delegate a third party to authorize a web request at the gate. All these application-layer heavy-liftings call for their dedicated computing resource — Ingress controller.

Standard Kubernetes Ingress

Standard Kubernetes Ingress is a better way to expose HTTP workloads, for its capability of controlling the traffic. In Kubernetes, an Ingress resource is an abstract resource type (i.e. an interface) to define traffic controlling rules. An Ingress controller is an implementation to satisfy the rules. It is the Ingress controller who accomplishes the given tasks like a hardworking gatekeeper. The tasks may include monitoring the incoming traffic, watching for rule updates, and executing the rules against the traffic.

With an Ingress resource taking on traffic control, we can revamp the Ingress path with a separation of concerns. We now only keep a single LoadBalancer Service object in place, with a bigger role. It serves as a common gateway for all ingress traffic destined to different workloads inside of the cluster. This task is critical and involving. Luckily, we can delegate the management of the Service object to an ingress controller. The shared load balancer appliance empowers ingress controller to scrutinize and route the incoming requests on their way into the cluster. Vendors of Ingress controller supports a wide range of routing capabilities based on attributes such as URL path and SNI. On the other side, the tenant workloads now only need to expose themselves within the cluster, expecting the Ingress controller to route requests their ways. Their lives become easier. Service objects with ClusterIP type fulfill the need.

Service Object vs Standard Ingress

With Ingress resource, we consolidate multiple open gates into a single shared gate. We thereby funnel all ingress traffic through this gateway, and intensively monitor and control the passing traffic. The load balancer Service object paves the path for entrance. The Ingress controller acts as the security guard in the way. And the Ingress resource resembles the “work instruction” for the guard. This pattern has many benefits. It reduces load balancer costs. It centralizes many points of configuration into a single one. Moreover, it helps define the boundary of responsibilities between operating teams. In a multi-tenant model, developing applications (e.g. billing service) is the responsibility of the tenant (e.g. developer of billing feature). They need not engineer how to expose their Services externally. The Ingress, its implementation, and routing configurations, are all considered infrastructure services for all tenants. They fall under the responsibility of the platform team.

Upon cluster creation, the platform team is expected to install the required Ingress controller. In some cases, the tenants may need multiple instances of Ingress, with multiple instances of Ingress controllers. It is again the responsibility of the platform team to install all the Ingress controllers and correctly reference them from each Ingress resource object.

The platform team also assumes the role of selecting the Ingress controller implementation for their tenants. Kubernetes only defines standard interface in its native Ingress resource, leaving the controller implementation up to each technology developer. No matter which developer, their designs generally follow the controller pattern. As an example, check out this document for how NGINX implements its Ingress controller. The role of an Ingress controller is similar to a reverse proxy on a web server. Kubernetes has compiled a list of implementation providers. From the list, we can find controllers that are:

• made by traditional players in application delivery controller (ADC) solutions: F5, HA Proxy and Citrix;
• made by public cloud providers: Azure’s AGIC, GKE Ingress on GCP and ALB Ingress Controller on AWS
• based on NGINX: NGINX controller and Kong Ingress controller
• built on top of Envoy (a modern open-source proxy project): Contour, Ambassador, EnRoute, Gloo

Apart from third-party options, the Kubernetes community themselves maintain a decent implementation of Ingress controllers also on top of NGINX. One should not mix up this community-version of Ingress controller (dubbed as ingress-nginx) with the other similarly named implementation by F5 team, known as the NGINX version. For more on Ingress controller options, LearnK8s website maintains a comprehensive table as community’s best effort. So take the information with a grain of salt.

The standard Ingress API took five years to graduate. As it fanned out, many came across its limitations. First, the standard Ingress resource doesn’t tackle non-HTTP services at all. To expose arbitrary TCP ports, the Ingress documentation suggests falling back on Service object with LoadBalancer type, sending us back to square one for lack of controlling capability. Second, even with HTTP workload, the routing rules that a standard Ingress resource can express is still simple. Features beyond the bare bones demand chunks of annotations only understandable by their provider. Moreover, Ingress resource centralizes the routing configuration for all tenants, whereas in the real-life operations, individual tenants oftentimes want to manage their own routing from their individual namespaces.

Ingress CRD

The native Kubernetes Ingress serves as a standard API schema. It does so at the cost of extensibility: developers have to sacrifice the flexibility to expose advanced features in their own interfaces. The workaround with proprietary annotations is cumbersome. Along with the annotation sprawl comes deteriorating maintainability. Since the inception of standard Ingress, a number of Ingress controller developers (e.g. Gloo, Contour, Traefik and HAProxy) have echoed each other on the pain points of massive annotations and shared the motives of switching to CRDs (Custom Resource Definitions).

Some took a step forward with their Ingress CRD products. As a result, their users can describe their Ingress as custom resources (CR), with developer-defined attributes. Take Traefik Labs for example, they have CRDs named IngressRouteTCP and IngressRoute. The former provides routing for any TCP traffic, a missing capability in standard Ingress. The latter builds a number of features for web traffic beyond the baseline routing capabilities, such as delegating routing configuration to tenants, and referencing secrets from different namespaces. These CRs work along with other Traefik CRs such as Middleware and TLSOption to deliver a complete set of features in their product offering.

In the Ingress CRD family, we can base our selection on any desired traffic management features as long as there are providers that support them. There are many features to look for. The common ones to consider are:

• Additional protocol support: HTTP/2, gRPC, TCP, WebSocket
• Dynamic configuration update without traffic interruption
• High Availability, throughput, load balancing algorithm, health check, sticky session
• Traffic Control: rate limiting, retry, circuit breaking, request rewrite, traffic splitting, traffic mirroring, canary and blue-green release
• Compatibility with service mesh

Before shopping around, the platform team should survey their stakeholders and lay out the detailed features their tenants need. On the vendor side, many contenders in the Ingress controller business also expand their footprints in the CRD market. Here is a list of some of these developers and the names of their routing CRDs:

• Traefik Labs: IngressRoute, IngressRouteTCP
• Kong: KongIngress
• F5 Nginx: VirtualServer and VirtualServerRoute
• Countour: HTTPProxy
• Gloo Edge by Solo: Gateway, Virtual Service
• Ambassador: Host and Mapping
• Istio: Gateway ,VirtualService and DestinationRule

To clarify compatibility with service mesh, Istio’s Gateway is a good example. Istio’s Gateway is the Ingress CRD provided as part of Istio service mesh. The Gateway CRD couples with other Istio CRDs to enable service mesh features such as traceability. If it is a hard requirement that your cluster run Istio as service mesh, then its Gateway CRD comes naturally as your choice of Ingress. Even though other Ingress controller or CRD might work, Istio Ingress Gateway features a unified configuration experience on Istio service mesh. That being said, Linkerd, another service mesh provider, holds the opposite design philosophy: their developers deliberately chose not to implement their own Ingress CRDs. Instead, they aim to make their product compatible with a range of commonly used Ingress controllers or CRDs.

These CRDs are so flexible that their use cases extend well beyond serving web clients. For example, many configure them into API gateways, the core component in the API Gateway pattern in microservice architecture. The use of ingress CRD as API Gateway is so widespread, that the term Gateway becomes better perceived than Ingress as the generic term to reflect what these feature-rich CRDs do. As the list above suggests, some CRD developers already brand their products as “Gateways” to stand distinguished from the Ingress controller family.

Gateway API

CRDs facilitate feature development, but their ecosystem has grown to a point where many vendors implement different CRDs with fundamentally similar features, a sign of fragmentation according to the governing community (SIG). The SIG Network community took the initiative to drive standardization using Gateway APIs. This initiative is well explained in this article, and demonstrated in KubeCon Europe 2021.

The Gateway API open-source project introduces a few more resource types, such as GatewayClass, Gateway, HTTPRoute and TCPRoute. The goal is to define expressive, extensible and role-oriented interfaces for gateway features, yet still leave it to different developers to plug their technology into Kubernetes. The Gateway API, once named “Service API” prior to February 2021, allows vendors to expose advanced features such as traffic splitting, RBAC, and TCP routing. To users, the Gateway API abstracts away the implementation details so it becomes easier to switch between technologies, thereby reducing vendor lock-in. In July 2022, the Gateway API project released v0.5.0 and graduated to beta.

The journey from standard Ingress to Gateway API demonstrates a swing in direction: the standard Ingress was an initial hasty standardization, leading to an unavoidable fragmentation with all kinds of Ingress CRDs. The Gateway API on the horizon signals a prudent effort to re-standardize it all, with a balance between uniformity and extensibility.

Competition already started in this arena. Most notably, the Envoy team introduced the Envoy Gateway project in May 2022, consolidating contour and Emissary projects, as an endorsement of Gateway API. From a production readiness perspective however, most of the Gateway API implementations are still in early stage of development. Despite of the ambitious vision, Gateway APIs have yet to reach the maturity level for production use.

Recommendations

Service object lacks traffic control and we should avoid using it alone. The futuristic Gateway API has yet to mature, leaving standard Ingress and Ingress CRDs the only two feasible options to control ingress traffic. At the time of writing, the recommendation is:

• If the requirement is undefined, start your POC with a standard Ingress controller.
• As you discover specific requirements such as routing config delegation, TCP routing, and minimizing annotation, explore the Ingress CRD market; but be wary of technology lock-in.
• If you have a requirement for service mesh, then the choice should consider the context of compatibility with service mesh.

Last but not least, watch for the latest progress of the Gateway API project, and develop a strategy for vendor neutrality.

Update from Dec 2023

The Gateway API project came to General Availability in Oct 2023. As a result, I would encourage reader to start venturing Gateway API.