Experiment: K8s Pod as an Internet Egress Network Appliance

Abstract Geometry with Dalle2, trying to show Pods and connectivity. Not sure I see it.

What a dry title. I promise it’s more interesting than it sounds.

Sometimes networks get crazy, due to: actually complicated use cases, bad initial planning and spaghettification over time, or for experimental purposes. This article falls into the experimental category, but I’ll let you be the judge.

The Stage

This idea came about when I was migrating services running in a docker compose setup to my local K3s cluster. A fairly easy task, but I stumbled upon a couple services that used network_mode to share a network with a VPN service I was running for additional security. I grappled with the question “how do I achieve this same functionality within a Kubernetes environment”?

Some context for the less avid docker compose folks out there (if such a thing exists), setting the network_mode on a service binds it’s network traffic through a docker defined network on another service. If Service A is a VPN service, and Service B is bound by network_mode to Service A, all traffic flowing from Service B will route through the VPN provided by Service A.

This idea of routing network traffic from one service to another is not as easy in Kubernetes, and I couldn’t (by a quick search) find a native way to approach this.

Yes, I could have setup a service mesh with a tool like Istio but that seemed like overkill and I wanted to find my own solution. Sometimes finding your own solution deepens knowledge in areas we are lacking.

The Execution

We can just add network_mode to the Pod spec and be good to go, right? I wish…

VPNs all the way down

The first step was to deploy a Pod that could be used as the network appliance. I used the nordvpn container for this purpose. Here is a sample of my deployment manifest:

apiVersion: v1
kind: Pod
metadata:
  name: nordvpn
  labels:
    app: nordvpn
spec:
  containers:
  - image: ghcr.io/bubuntux/nordvpn
    imagePullPolicy: IfNotPresent
    name: nordvpn
    env:
      - name: TOKEN
        value: <REDACTED>
      - name: DNS
        value: "1.1.1.1"
      - name: NET_LOCAL
        value: "10.42.2.0/24"
    securityContext:
      capabilities:
        add: ["NET_ADMIN", "NET_RAW"]
  restartPolicy: Always

Running kubectl get pods -o wide shows me the IP address associated with this now running nordvpn pod. In my home lab environment that IP is 10.42.2.66.

I’ve also included the environment variable NET_LOCAL with the value 10.42.2.0/24 which is the CIDR for the pods in my cluster. This causes the nordvpn container to setup IP forwarding and iptables properly to create a router through the VPN for traffic originating from those IPs.

Routes on routes

The next, more difficult, layer was to find out how to configure another Pod to route all internet traffic through that nordvpn Pod.

After chatting with ChatGPT and doing a bit of research, I decided to use the ip command in an Ubuntu container and manually set the default route.

Quick spec for a basic Ubuntu pod:

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu
  labels:
    app: ubuntu
spec:
  containers:
  - image: ubuntu
    command:
      - "sleep"
      - "604800"
    imagePullPolicy: IfNotPresent
    name: ubuntu
    securityContext:
      capabilities:
        add: ["NET_ADMIN", "NET_RAW"]
  restartPolicy: Always

This spec gives me a sleeping Ubuntu Pod I can exec into and run tests with the ip command.

After deploying, connect to the pod like kubectl exec -it ubuntu -- /bin/bash .

Here is the log of commands I ran to configure the proper routing.

# Gotta fetch updates
apt update

# Install the ip tool
apt install iproute2 -y

# Delete the default route
ip route del default

# Add a new default route targeting the nordvpn Pod
ip route add default via 10.42.2.66

I wasn’t convinced this was working. So I installed curl and ran curl https://ipinfo.io/ip to fetch the public IP the Ubuntu Pod. Sure enough, it was 185.158.242.198 which is the end node of a UK based nordvpn tunnel.

It works! But for now only manually.

I wrapped up this test by defining the IP routing process as part of the postStart lifecycle hook of the Ubuntu pod. That spec is:

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu-vpn-test
  labels:
    app: ubuntu-vpn-test
spec:
  containers:
  - image: ubuntu
    command:
      - "sleep"
      - "604800"
    imagePullPolicy: IfNotPresent
    name: ubuntu-vpn-test
    env:
      - name: VPN_IP
        value: "10.42.2.66"
    securityContext:
      capabilities:
        add: ["NET_ADMIN", "NET_RAW"]
    lifecycle:
      postStart:
        exec:
          command: 
          - "/bin/bash"
          - "-c"
          - "apt update && apt install iproute2 -y && ip route del default && ip route add default via $VPN_IP"
  restartPolicy: Always

Notice I moved the VPN_IP to an environment variable just to make the script easier.

With this strategy in mind, I can now deploy any number of Pods and egress them all through that nordvpn Pod.

Forward Thoughts

Clearly this doesn’t scale well, for a number of reason.

• Single nordvpn Pod creates a single point of failure.
• If the nordvpn Pod is recreated, changing it’s IP address all other pods need to be updated.
• Needing to manually add that lifecycle hook script to all Pods would be a pain.
• etc.

But it is kind of a fun solution.

A future test might be to write a Kubernetes operator that:

• Watches Pod deployment, and given a specific label can add the lifecycle hook automatically.
• Watches the nordvpn Pod for a changed IP address and update the other Pods accordingly.
• etc.

Regardless, test concluded with a working solution. Time to take the dogs for a walk.

Let me know if you have any questions about the above, I love chatting about this stuff.