AWS EKS and Pod sizing per Node considerations

ismail yenigül

Follow

Feb 22, 2019 · 3 min read

From https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html

Amazon EKS supports native VPC networking via the Amazon VPC CNI plugin for Kubernetes. Using this CNI plugin allows Kubernetes pods to have the same IP address inside the pod as they do on the VPC network.

Note: This limitation is exist only if you are using the default Amazon VPC CNI plugin for Kubernetes. There is no problem if you using other CNI plugins

This is really a great feature. But there is a one import limitation we will encounter if you want to run so many Pods in a EC2 worker node. Whenever you deploy a Pod in the EKS worker node, EKS creates a new IP address from VPC subnet and attach to the instance.

You can check how many ethernet interfaces and max IP per interface from https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI

Here is the formula for max Pods numbers.

Max Pods = Maximum supported  Network Interfaces for instance type ) * ( IPv4 Addresses per Interface ) - 1

For example, if you have a t3.medium instance which support max 3 ethernets and 6 IPs per interface. You can create only 17 pods including the kubernetes internal Pods, Because One IP is reserved for nodes itself.

3 * 6 - 1 = 17

You can see run the following commands to see how many Pods are running.

# kubectl get pods — all-namespaces | grep -i running | wc -l
17

If you want to deploy more pods in the EC2 worker node, you should deploy larger EC2 instance type.

Note: I faced this problem when I was deploying alb-ingress-controller Pod.

# kubectl logs -n kube-system $(kubectl get po -n kube-system | egrep -o alb-ingress[a-zA-Z0–9-]+)
...
Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “ae9a4d174c7c1251ddddcc2304a81520162d5dd788e04448d545e0980c14c816” network for pod “alb-ingress-controller-78cf6874c9-qjk6x”: NetworkPlugin cni failed to set up pod “alb-ingress-controller-78cf6874c9-qjk6x_kube-system” network: add cmd: failed to assign an IP address to container

Ismail YENIGUL

Devops Engineer

Follow us on Twitter 🐦 and Facebook 👥 and join our Facebook Group 💬.

To join our community Slack 🗣️ and read our weekly Faun topics 🗞️, click here⬇

If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇