IP exhaustion in EKS

Holaa!! I’m back with one more story , so last time I am having a conversation with a DevOps engineer and during the conversation he told he is having an IP exhaustion with their EKS clusters and they want to fix it and then I realized this could be a good article for who has recently started k8s or who face this issue but don’t have right solution yet.

Problem:

In most corporates we don’t have control on the subnet provisioning and one common issue we face is when we ask the networking guys for /16 subnet they will be coming back with a /26 and adjust for it :P (Not in every corporate though , I mean I am talking abt the rich folks:P) . This works well and good till we are using limited ec2 instances with docker on it and when they are getting squeezed we request for new ones . And in docker anyway containers use docker networking , But the game changes completely when u move to k8s , In k8s every thing more or less ends up as a pod and every pod consumes an IP and that too from the subnets that you guys attached and slowly u see your /26 cant even sustain a small app on EKS because any pod that gets created will just eat an IP and sometimes when u need your cluster to scale you observe your new node is not left with an IP and you realize how evil things at k8s is and u also understand how evil your networking guys too are :P .So enough of these but what’s the resolution? Ok Ok don’t worry we are going there .

So essentially EKS folks too observed this common issue and they brought in Secondary CNI(Container Network Interface) . But what is this ?

In an EKS cluster, a secondary CNI is an additional networking plugin that can be used in conjunction with the default CNI plugin, which is usually Amazon VPC CNI.

So how is this helpful? To put it simple u want to tell k8s that please don’t consume your10. range for pods and don’t act as as bottleneck for scaling rather you promise k8s that you will give it a different subnet and it could use that for its pod networking rather than using the node IP’s .Doesn’t it sound heaven after all the issues you faced! So what do we do to get that done .

But there is a catch for secondary CNI’s only CIDR with range 100.64.0.0/10 or 198.19.0.0/16 (find reference here)only and on top of that you could also add Security rules using security groups on pods too. So how to do it real time?

Lets Go ahead with the procedure then

1. Login to your EKS cluster and set the context to your cluster
2. Make sure you aws vpc cni add-on is on your cluster if no go to console and add it from add-ons It gets installed by default currently

3. Execute the following command

kubectl set env daemonset aws-node -n kube-system AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true

4. Execute this command too

kubectl set env daemonset aws-node -n kube-system ENI_CONFIG_LABEL_DEF=failure-domain.beta.kubernetes.io/zone

5. retrieve the security group of your cluster using the below command

cluster_security_group_id=$(aws eks describe-cluster --name $cluster_name --query cluster.resourcesVpcConfig.clusterSecurityGroupId --output text)

6. Now apply the ENICONFIG based on the availability zones(az’s) your cluster is in.

#az1.yaml
----
apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata: 
  name: <az_1>
spec: 
  securityGroups: 
    - <cluster_security_group_id>
  subnet: <subnetid-1> 

----
#az2.yaml
apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata: 
  name: <az_2>
spec: 
  securityGroups: 
    - <cluster_security_group_id>
  subnet: <subnetid-2>

---
#Lets say you az is us-west and wanna create files here is a sample

apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata: 
  name: eu-west-1a
spec: 
  securityGroups: 
    - sg-873498jdjdks
  subnet: subnet-jhdghjdfs34

----

apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata: 
  name: eu-west-1b 
spec: 
  securityGroups: 
    - sg-32323jsdsd
  subnet: subnet-dfdfre98d3

Now apply these configuration files

kubectl apply -f az1.yaml
kubectl apply -f az2.yaml

Now do this to verify

kubectl get ENIConfigs

You should see output like this 
NAME         AGE
us-west-1a   117s
us-west-1b   105s

So now you need to do the following

Scale down the nodes to ‘0’.

aws autoscaling set-desired-capacity  --auto-scaling-group-name <asg-group for the concerned nodegroups>--desired-capacity 0 --region=eu-west-1

Remember you cant just go and do this on prod cluster make sure you are

1. Check whether you have available nodes that are using the custom networking feature.
2. Cordon and drain the nodes to gracefully shut down the pods. For more information, see Safely Drain a Node in the Kubernetes documentation.
3. Now terminate the nodes.

4. If its a non-prod node or a new cluster setup scale down and scale them back to the previous node size.

aws autoscaling set-desired-capacity  --auto-scaling-group-name <asg-group for the concerned nodegroups>--desired-capacity <previous size>--region=eu-west-1

5. Now do the below check and you should see core-dns pods being spawned in the 100.64.0.0/16 range like expected

kubectl get pods -n kube-system -o wide

NAMESPACE     NAME                       READY   STATUS    RESTARTS    AGE         IP                   NODE                                          NOMINATED NODE   READINESS GATES

kube-system   coredns-8978732ccd-fgddr    1/1     Running    0          5m      100.64.18.23     ip-10-168-0-92.us-west-1.compute.internal       <none>           <none>
kube-system   coredns-89738432sbc-jkha9   1/1     Running    0          5m      100.64.11.24     ip-10-168-0-92.us-west-1.compute.internal      <none>           <none>

Hurray!! From here on you are not gonna face the IP exhaustion at subnets and no more “network: add cmd: failed to assign an IP address to container” error.

So lets say you want cloud agnostic or dont intend to use aws cni’s what shld we use then , dont worry there are alternatives too flannel and calico . They are quite famous alternatives too.

But do you think your life is a path of roses!! No yaar that’s not gonna happen :D WHY?

Because there is one more issue you need to deal with IP’s and Nodes!!! What is that ? Follow me on next article that’s a great hook I would say :P

In the meanwhile added all the references down here

Reference:

Addressing IPv4 address exhaustion in Amazon EKS clusters using private NAT gateways | Amazon Web…

Working with the Amazon VPC CNI plugin for Kubernetes Amazon EKS add-on

Amazon VPC CNI for EKS

Amazon VPC CNI - EKS Best Practices Guides

Use multiple CIDR ranges with Amazon EKS

Custom Networking - EKS Best Practices Guides

Tutorial: Custom networking