How to plan/manage IP address space in GKE?
I have seen many GKE cluster deployment failures due to IP address exhaustion. In this blog post I will try to explain how you can plan your IP address space in advance before creating the clusters. Rest of the blog assumes you are creating private and VPC native GKE clusters, which is most widely used by organisations across the industries.
GKE needs following IP address ranges:
- Nodes Range: This will be used to allocate IPs to nodes in the cluster.
- Pods Range: This will be used to allocate IPs to PODs in the cluster.
- Services Ranges: This will be used to allocate IPs(Cluster IPs) to Services in the cluster.
- Control Plane Range: This will be used to allocate IPs to the control plane nodes.
- Internal Load Balancer Range: This will be used to allocate the IPs to internal load balancers. This is optional and if you don’t use this then Internal Load Balancers will get IPs from Nodes Range.
To find the appropriate ranges, answers to following questions will help us.
- How many nodes do we need in the cluster? Let’s say we need 50 nodes in the cluster.
- How many pods will be scheduled/created on the nodes i.e. what is pod density? Let’s say we will be creating 30 pods per node. If we don’t choose any custom value then default is 110.
- How many services do we need? Let’s say we need 200 services.
- How many ILB backed services do we need? Let’s say we need 100 ILB backed services.
Control Plane Range
Control plane range must be RFC1918 range with /28 CIDR. RFC1918 defines below CIDR ranges as private IP ranges:
10.0.0.0–10.255.255.255 (10/8 prefix)
172.16.0.0–172.31.255.255 (172.16/12 prefix)
192.168.0.0–192.168.255.255 (192.168/16 prefix)
For example 10.0.0.0/28 can be one such range which can be used for control plane subnets. This provides 16 IPs and will be used by GKE control plane nodes.
Nodes Range
Since we need 50 nodes, /26 CIDR range will give us 64 addresses which can accommodate our required number of nodes. I am assuming we will be using different CIDR ranges for ILBs.
For example 10.10.0.0/26 can be one such range for the node’s CIDR subnet.
Pods Range
In Kubernetes each node carves out a CIDR range from pods CIDR range to allocate IPs to pods scheduled on the node. Since we need 30 pods per node we need a subnet which can allocate IPs to 60 pods on each node. Why 60 not 30, is because we need to take care of rolling updates of the pods into consideration as well. During rolling updates two different versions of the same pod can exist which need double address space.
So each node needs /26 (it provides 64 IPs) address space for pods. We have 50 such nodes, which means we need address space which can allocate one /26 address space to each of the 50 nodes.
We need a address space which can allocate 50(nodes) * 60(pods per node) = 3000 IPs i.e. we need a /20 CIDR range. /20 provides us 4096 IPs which is more than the required 3000 IPs. For example 10.114.0.0/20 can be one such range.
Services Range
Since we need 200 Services, /24 CIDR provides us the required IPs. For example 10.115.0.0/24 can be one such range.
ILB Range
Since we need 200 Services, /25 CIDR provides us the required IPs. For example 10.116.0.0/25 can be one such range.
For more details please check:
https://cloud.google.com/kubernetes-engine/docs/best-practices/networking#ip-address-mgmt
https://cloud.google.com/blog/products/containers-kubernetes/ip-address-management-in-gke
https://cloud.google.com/kubernetes-engine/docs/how-to/flexible-pod-cidr
