Kubernetes bypass routing traffic to destination with service type load balancer inside the cluster

7 min readJul 1, 2024

Introduction

In this article, I discuss a known issue in Kubernetes versions before 1.30. In older Kubernetes releases, kube-proxy would intercept traffic destined for the IP address associated with a Service of type LoadBalancer. On Linux, kube-proxy in iptables mode would redirect packets directly to the endpoint, while in IPVS mode, kube-proxy would configure the load balancer’s IP address on one interface of the node.

The Issue

Imagine you want to implement a solution inside Oracle Container Engine for Kubernetes using an Oracle Cloud Infrastructure (OCI) load balancer as the front end for the services. Here’s the scenario:

An application is deployed in a pod and exposed as a ClusterIP service.
An ingress controller inside the cluster routes traffic for a specific host and URI to the application service.
The ingress controller itself is of type LoadBalancer, using an OCI load balancer, and SSL termination is handled by the OCI load balancer.

When a request is initiated from a virtual machine outside the Kubernetes cluster (the blue path), the request (https://<host>:443/<application URI>) successfully reaches the target pod. However, when the same request is initiated from a pod inside the cluster, the host name resolves to the load balancer IP address. Due to the bug mentioned earlier, the request bypasses the load balancer (the red path) and we receive a 404 error because SSL routing is not defined in the ingress controller for the application pod.

Implementation

Ingress Controller (Traefik in my implementation) service

kubectl get svc -n traefik

NAME      TYPE           CLUSTER-IP    EXTERNAL-IP         PORT(S)                      AGE
traefik   LoadBalancer   10.96.8.213   <Load Balancer IP>  80:31679/TCP,443:31365/TCP   76d

Application Pod (Nginx server)

kubectl get pod

NAME         READY   STATUS             RESTARTS   AGE
nginx-pod    1/1     Running            0          3d1h

Application Service

kubectl get svc

NAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
nginx-service   ClusterIP   10.96.138.130   <none>        80/TCP              73m

Ingress Route definition

kubectl describe ingressroute  nginx-ingressroute

Name:         nginx-ingressroute
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  traefik.containo.us/v1alpha1
Kind:         IngressRoute
Spec:
  Entry Points:
    web
  Routes:
    Kind:   Rule
    Match:  Host(`nginx.cloud`) && PathPrefix(`/`)
    Services:
      Name:  nginx-service
      Port:  80

The load balancer listens on port 443 using a self-signed certificate and terminates SSL on the load balancer.

Testing

From Outside the Cluster

curl -v -k -H "HOST: nginx.cloud" https://<LB IP>/

*   Trying <LB IP>:443...
* Connected to <LB IP> (<LB IP>) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384 / [blank] / UNDEF
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: C=AU; ST=VIC; L=Melbourne; O=Oracle; OU=COE; CN=nginx.cloud; emailAddress=contact@nginx.cloud
*  start date: Jun 28 02:16:54 2024 GMT
*  expire date: Oct  1 02:16:54 2026 GMT
*  issuer: C=AU; ST=VIC; L=Melbourne; O=Oracle; OU=COE; CN=nginx.cloud; emailAddress=contact@nginx.cloud
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* using HTTP/1.x
> GET / HTTP/1.1
> Host: nginx.cloud
> User-Agent: curl/8.6.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Content-Length: 615
< Content-Type: text/html
< Date: Fri, 28 Jun 2024 02:48:42 GMT
< Etag: "6655da96-267"
< Last-Modified: Tue, 28 May 2024 13:22:30 GMT
< Server: nginx/1.27.0
<
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
* Connection #0 to host <LB IP> left intact

This command yields a successful response from the Nginx pod.

From Inside the Cluster

curl -v -k -H "HOST: nginx.cloud" https://<LB IP>/

*   Trying <LB IP>:443...
* Connected to <LB IP> (<LB IP>) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / x25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=TRAEFIK DEFAULT CERT
*  start date: Jun 28 01:58:02 2024 GMT
*  expire date: Jun 28 01:58:02 2025 GMT
*  issuer: CN=TRAEFIK DEFAULT CERT
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://<LB IP>/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: nginx.cloud]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: nginx.cloud
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/2 404
< content-type: text/plain; charset=utf-8
< x-content-type-options: nosniff
< content-length: 19
< date: Fri, 28 Jun 2024 02:50:56 GMT
<
404 page not found
* Connection #0 to host 192.9.177.76 left intact

This command results in a 404 error because the request lands on the ingress controller’s SSL port, for which no route is configured (and Load balancer has been bypassed and traffic directly routed from Ingress controller to the pod) .

Solution

This issue is resolved in Kubernetes 1.30. For older versions, a workaround is to add .status.loadBalancer.ingress.hostname to the Ingress Controller service.

Current Ingress Controller Service Definition

kubectl get svc traefik -n traefik -o yaml

apiVersion: v1
kind: Service
  name: traefik
  namespace: traefik
spec:
  allocateLoadBalancerNodePorts: true
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ports:
  - name: web
    nodePort: 31679
    port: 80
    protocol: TCP
    targetPort: web
  - name: websecure
    nodePort: 31365
    port: 443
    protocol: TCP
    targetPort: websecure
  selector:
    app.kubernetes.io/instance: traefik-traefik
    app.kubernetes.io/name: traefik
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: <LB IP>

Edited Service Definition

Edit the service to add the hostname:

kubectl edit --subresource=status svc traefik -n traefik

Add the following under status.loadBalancer.ingress:

status:
  loadBalancer:
    ingress:
    - hostname: nginx.cloud

Testing After the Workaround

Executing the same curl command from inside the cluster now yields a successful response:

curl -v -k -H "HOST: nginx.cloud" https://<LB IP>/

*   Trying <LB IP>:443...
* Connected to <LB IP> (<LB IP>) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384 / secp256r1 / rsaEncryption
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: C=AU; ST=VIC; L=Melbourne; O=Oracle; OU=COE; CN=nginx.cloud; emailAddress=contact@nginx.cloud
*  start date: Jun 28 02:16:54 2024 GMT
*  expire date: Oct  1 02:16:54 2026 GMT
*  issuer: C=AU; ST=VIC; L=Melbourne; O=Oracle; OU=COE; CN=nginx.cloud; emailAddress=contact@nginx.cloud
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/1.x
> GET / HTTP/1.1
> Host: nginx.cloud
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Content-Length: 615
< Content-Type: text/html
< Date: Fri, 28 Jun 2024 03:06:37 GMT
< Etag: "6655da96-267"
< Last-Modified: Tue, 28 May 2024 13:22:30 GMT
< Server: nginx/1.27.0
<
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
* Connection #0 to host <LB IP> left intact

Conclusion

This workaround is not permanent and might be reverted by service updates, requiring re-application. I hope this article helps resolve similar issues in your environment.

References

Kubernetes 1.29 Feature: LoadBalancer IP Mode Alpha