Open in app

Sign in

Write

Sign in

Cilium: Installing Cilium in EKS with no Kube-Proxy

Amit Gupta
7 min readMar 21, 2024

--

Source: docs.cilium.io

☸ ️Introduction

kube-proxy is a component of Kubernetes that handles routing traffic for services within the cluster. There are two backends available for Layer 3/4 load balancing in upstream kube-proxy - iptables and IPVS.

Need for Kube-Proxy:

IPtables and Netfilter are the two foundational technologies of kube-proxy for implementing the Service abstraction. They carry legacy accumulated over 20 years of development grounded in more traditional networking environments that are typically far more static than your average Kubernetes cluster. In the age of cloud native, they are no longer the best tool for the job, especially in terms of performance, reliability, scalability, and operations.

Cilium to the rescue

Cilium’s kube-proxy replacement offers advanced configuration modes to cater to your specific needs. Features like client source IP preservation ensure that your service connections remain intact, while Maglev Consistent Hashing enhances load balancing and resiliency. With support for Direct Server Return (DSR) and Hybrid DSR/SNAT modes, you can optimize traffic routing and improve performance.

🎯Goals & Objectives

In this article you will learn how the EKS-managed kube-proxy DaemonSet can also be disabled entirely to install Cilium as the CNI on an EKS cluster with a managed nodegroup.

Pre-Requisites

  • You should have an AWS subscription.
  • Install kubectl
  • Install Helm
  • The following EC2 privileges are required by the Cilium operator in order to perform ENI creation and IP allocation.
  • Install eksctl
  • Install awscli
  • Cilium CLI: Cilium provides Cilium CLI tool that automatically collects all the logs and debug information needed to troubleshoot your Cilium installation. You can install Cilium CLI for Linux, macOS, or other distributions on their local machine(s) or server(s).

Let’s get going

kube-proxy maintains network rules on each Amazon EC2 node. It enables network communication to your Pods. The self-managed or managed type of this add-on is installed on each Amazon EC2 node in your cluster, by default.

Create a new cluster and disable kube-proxy.

  • Create sample configuration file and name it eks-config_ipv4.yaml
    -Note- you can create an EKS cluster from the command line utility as well.
    -managedNodeGroups should be tainted with node.cilium.io/agent-not-ready=true:NoExecute to ensure application pods will only be scheduled once Cilium is ready to manage them.
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: cluster1
  region: ap-southeast-2
  version: "1.27"

managedNodeGroups:
- name: ng-1
  desiredCapacity: 2
  privateNetworking: true
  # taint nodes so that application pods are
  # not scheduled/executed until Cilium is deployed.
  # Alternatively, see the note below.
  taints:
   - key: "node.cilium.io/agent-not-ready"
     value: "true"
     effect: "NoExecute"
  • Create the EKS cluster using the configuration file created in the previous step
eksctl create cluster -f eks-config_ipv4.yaml
  • Once the cluster is up and running; ensure that all the nodes are up and in a healthy state.
 kubectl get nodes -o wide
NAME                                                STATUS   ROLES    AGE     VERSION               INTERNAL-IP      EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION                  CONTAINER-RUNTIME
ip-192-168-105-179.ap-southeast-2.compute.internal  Ready    <none>   4d17h   v1.28.5-eks-5e0fdde   192.168.105.179   <none>        Amazon Linux 2   5.10.210-201.852.amzn2.x86_64   containerd://1.7.11
ip-192-168-97-26.ap-southeast-2.compute.internal    Ready    <none>   4d17h   v1.28.5-eks-5e0fdde   192.168.147.228   <none>        Amazon Linux 2   5.10.210-201.852.amzn2.x86_64   containerd://1.7.11

Delete the kube-proxy daemonset and configmap

  • For existing installations with kube-proxy running as a DaemonSet, remove it by using the following commands.
kubectl -n kube-system delete ds kube-proxy

kubectl -n kube-system delete cm kube-proxy
  • Ensure that kube-proxy is no longer running as a daemonset.
kubectl get ds -A
NAMESPACE     NAME       DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                     AGE
kube-system   aws-node   0         0         0       0            0           io.cilium/aws-node-enabled=true   8d
kube-system   cilium     2         2         2       2            2           kubernetes.io/os=linux            17h

kubectl get cm -A
NAMESPACE         NAME                                                   DATA   AGE
default           kube-root-ca.crt                                       1      8d
kube-node-lease   kube-root-ca.crt                                       1      8d
kube-public       kube-root-ca.crt                                       1      8d
kube-system       amazon-vpc-cni                                         2      8d
kube-system       aws-auth                                               1      8d
kube-system       cilium-config                                          105    17h
kube-system       coredns                                                1      8d
kube-system       extension-apiserver-authentication                     6      8d
kube-system       kube-apiserver-legacy-service-account-token-tracking   1      8d
kube-system       kube-root-ca.crt                                       1      8d

Remove kube-proxy iptable entires from each node

  • With root permissions ensure that iptable entries pertinent to kube-proxy are removed.
    - This is done to clear out AWS-VPC-CNI related rules.
iptables-save | grep -v KUBE | iptables-restore

iptables-save | grep -E -v 'AWS-SNAT-CHAIN|AWS-CONNMARK-CHAIN' | iptables-restore

Note- Updated as of 27/06/2024 with inputs from Isovalent team member- Scott Lowe.

If you provision an EKS cluster without the AWS-VPC-CNI plugin, then you don’t need to do the above step.

Install Cilium

  • You can install Cilium in either ENI mode or Overlay mode on an EKS cluster.
  • In case of ENI mode, Cilium will manage ENIs instead of VPC CNI, so the aws-node DaemonSet has to be patched to prevent conflict behavior.
kubectl -n kube-system patch daemonset aws-node --type='strategic' -p='{"spec":{"template":{"spec":{"nodeSelector":{"io.cilium/aws-node-enabled":"true"}}}}}'
  • Make sure you correctly set your API_SERVER_IP and API_SERVER_PORT by using:
kubectl cluster-info
  • Set up Helm repository
helm repo add cilium https://helm.cilium.io/
  • Install Cilium using helm
API_SERVER_IP=<your_api_server_FQDN>
API_SERVER_PORT=<your_api_server_port>

helm install cilium cilium/cilium --version 1.14.8 \
  --namespace kube-system \
  --set eni.enabled=true \
  --set ipam.mode=eni \
  --set egressMasqueradeInterfaces=eth0 \
  --set routingMode=native \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=${API_SERVER_IP} \
  --set k8sServicePort=${API_SERVER_PORT}
  • Validate that the Cilium agent is running in the desired mode
kubectl -n kube-system exec ds/cilium -- cilium status | grep KubeProxyReplacement
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
KubeProxyReplacement:    True   [eth0 192.168.105.179 (Direct Routing)]
  • Create a sample service (100 in number)
for x in {1..100}; do yq -i ' .metadata.name = "nginx-svc-'$x'" ' service.yaml | kubectl apply  -f service.yaml ;done 
service/nginx-svc-1 unchanged
service/nginx-svc-2 unchanged
service/nginx-svc-3 created
service/nginx-svc-4 created
service/nginx-svc-5 created
service/nginx-svc-6 created
service/nginx-svc-7 created
service/nginx-svc-8 created
service/nginx-svc-9 created
[]
service/nginx-svc-89 created
service/nginx-svc-90 created
service/nginx-svc-91 created
service/nginx-svc-92 created
service/nginx-svc-93 created
service/nginx-svc-94 created
service/nginx-svc-95 created
service/nginx-svc-96 created
service/nginx-svc-97 created
service/nginx-svc-98 created
service/nginx-svc-99 created
service/nginx-svc-100 created
  • Check the EKS nodes for any iptable entries related to kube-proxy
[root@ip-192-168-105-79 /]# iptables-save | grep -c KUBE-SVC
0
[root@ip-192-168-147-228 /]# iptables-save | grep -c KUBE-SEP
0

How can we ensure that kube-proxy is not installed post a Kubernetes version upgrade?

  • You can also optionally validate that kube-proxy is not installed as an add-on a subsequent kubernetes upgrade.
  • As you can see in this example below, the EKS cluster is upgraded from k8s version 1.27 to k8s version 1.29 and we don’t see kube-proxy being enabled as an add-on.
eksctl upgrade cluster --name cluster1 --region ap-southeast-2 --version 1.29 --approve

eksctl get nodegroup ng-1 --region ap-southeast-2 --cluster=cluster1
CLUSTER  NODEGROUP STATUS CREATED   MIN SIZE MAX SIZE DESIRED CAPACITY INSTANCE TYPE IMAGE ID ASG NAME     TYPE
cluster1 ng-1  ACTIVE 2024-03-10T16:13:06Z 2  2  2   m5.large AL2_x86_64 eks-ng-1-54c71491-cd91-5c20-0e4c-bb39fea7a7b7 managed

eksctl upgrade nodegroup \
  --name=ng-1 \
  --cluster=cluster1 \
  --region=ap-southeast-2 \
  --kubernetes-version=1.29
  • Validate that the Cilium agent is running in the desired mode
kubectl -n kube-system exec ds/cilium -- cilium status | grep KubeProxyReplacement
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
KubeProxyReplacement:    True   [eth0 192.168.177.57 (Direct Routing)]
  • Validate that kube-proxy is not present as a daemonset post the upgrade.
kubectl get ds -A
NAMESPACE     NAME       DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                     AGE
kube-system   aws-node   0         0         0       0            0           io.cilium/aws-node-enabled=true   8d
kube-system   cilium     2         2         2       2            2           kubernetes.io/os=linux            19h

kubectl get cm -A
NAMESPACE         NAME                                                   DATA   AGE
default           kube-root-ca.crt                                       1      8d
kube-node-lease   kube-root-ca.crt                                       1      8d
kube-public       kube-root-ca.crt                                       1      8d
kube-system       amazon-vpc-cni                                         2      8d
kube-system       aws-auth                                               1      8d
kube-system       cilium-config                                          105    19h
kube-system       coredns                                                1      8d
kube-system       extension-apiserver-authentication                     6      8d
kube-system       kube-apiserver-legacy-service-account-token-tracking   1      8d
kube-system       kube-root-ca.crt                                       1      8d

References

Known Issue (with Terraform)

  • Issue with EKS and Terraform modules when the kube-proxy add-on gets re-added.

Try out Cilium

  • Try out Cilium and get a first-hand experience of how it solves some real problems and use-cases in your cloud-native or on-prem environments related to Networking, Security or Observability.

🌟Conclusion 🌟

Hopefully, this post gave you a good overview of how to install Cilium on EKS in ENI or Overlay mode with no kube-proxy. Thank you for Reading !! 🙌🏻😁📃, see you in the next blog.

🚀 Feel free to connect/follow with me/on :

LinkedIn: linkedin.com/in/agamitgupta

Cilium
Ebpf
Elastickubernetesservice
Kubernetes
Kube Proxy

--

--

Amit Gupta

Written by Amit Gupta

63 Followers

Sr. TME @ Isovalent

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams