EKS Networking— VPC CNI Modes
Pod IP addressing in EKS
AWS EKS uses a simple approach for pod networking. It uses the IPs from the VPC for the Pod. The same is achieved by leveraging the VPC CNI plugin, a daemonset(aw-node) running in the cluster. VPC CNI plugin acts as a Bridge between K8s & AWS VPC.
This plugin leverages the fact that EC2 instances can have multiple network interfaces (ENI) and each ENI can have more than one IPs. The CNI plugin provisions secondary interfaces and adds multiple IPs to it. When a pod creation request comes it allocates an IP from the pool of IPs attached to secondary ENI. When the number of pods running on the node exceeds the number of addresses on a single ENI, the CNI backend starts allocating a new ENI. CNI Plugin exposes the below-mentioned configurations to control IP & ENI reservation.
To understand the above configuration visit my earlier article https://medium.com/nerd-for-tech/eks-networking-cni-457ae298b9e6
The plugin runs mainly in three modes.
In this mode, the secondary ENI reserves the IP address from the subnet.
In this mode, the secondary ENI maps to a subnet prefix and allocates IP to pods from the prefix. The prefix is of /28. This means we are increasing the number of IPs available to nodes by a factor of 16. Say for c5.large we can have 3 ENI and each can have 10 IPs. This gives us 27 usable IPs. But with prefixes, we will have 432 (3*9*16) IPs. This allows us in greater pod density.
Custom configuration mode
In this mode, CNI will use secondary CIDRs. Secondary CIDR ranges are an additional set of subnets deployed to the same availability zones as your existing private subnets. The only difference, these subnets use a secondary CIDR range, rather than the primary CIDR range attached to the VPC. In this mode, the worker node will get IP out of our primary CIDR range and pods will get IP out of our secondary CIDR.
In this blog, I tried to explain how IPs are assigned to Pods in EKS in different modes.