Migrate to Cilium from Amazon VPC CNI with Zero Downtime

CNI (Container Network Interface) is a fundamental framework for configuring and managing network resources and connectivities of containers, used by container runtimes.

There are a bunch of CNI implementations used by Kubernetes and Cilium is one of them with popular usage.

UPDATE: A terraform example of migrating steps in this post is added to terraform-examples/examples/cilium-cni-migration at main · rockc2020/terraform-examples (github.com).

Why Cilium CNI?

We are using AWS EKS clusters which have the Amazon VPC CNI installed by default. However, there are limitations of Amazon VPC CNI regarding security of cluster networking.

NetworkPolicy

The biggest drawback of Amazon VPC CNI for us is the lack of support for NetworkPolicy. NetworkPolicy can be used for isolating Pod network traffic in multi-tenant environments which controls ingress and egress traffic for tenant Pods.

Lots of alternative CNIs can enforce NetworkPolicy including Cilium, such as Calico.

Cross-Cluster NetworkPolicy

Kubernetes NetworkPolicy works well inside a single cluster but it’s not enough to control the traffic crossing Kubernetes…