Scheduling Kubernetes Pods on AWS
Provider-specific details influence the way you manage your Kubernetes applications. For example, AWS EBS volumes can’t be mounted from a different zone. If your pod needs a specific volume, it must be in the same zone as the volume.
To improve readability, I’ll use Koki Short, a simpler but equivalent syntax for Kubernetes manifests. Here’s a Pod configured specifically to run in us-east-1a
:
pod:
affinity:
- node: failure-domain.beta.k8s.io/zone=us-east-1a
containers:
- image: me/my-server
It uses Node Affinity (read about it here) to make the Pod run on a Node with the failure-domain.beta.k8s.io/zone
label us-east-1a
.
If we have other clusters in different regions, we can configure the Pod for us-west-1a
or even ap-southeast-1a
:
pod:
affinity:
- node: failure-domain.beta.k8s.io/zone=us-west-1a
containers:
- image: me/my-server---pod:
affinity:
- node: failure-domain.beta.k8s.io/zone=ap-southeast-1a
containers:
- image: me/my-server
It’s great that we can write Pod manifests that run in a specific zone, but we don’t want to maintain duplicated YAML. The manifests above are nearly identical — the only change is from us-east-1a
to us-west-1a
or ap-southeast-1a
.
Let’s factor these Pods into reusable components. For example, we can pull out the Node Affinity rule:
# node-us-west-1a.yaml
affinity:
- node: failure-domain.beta.k8s.io/zone=us-west-1a
And import it:
imports:
- affinity: ./node-us-west-1a.yaml
pod:
affinity: ${affinity}
containers:
- image: me/myserver
We can also pull out the rest of the Pod spec:
# myserver-pod.yaml
params:
- affinity: the Pod's affinities (for scheduling)
pod:
affinity: ${affinity}
containers:
- image: me/myserver
And import it:
imports:
- affinity: ./node-us-west-1a.yaml
- pod: ./myserver-pod.yaml
params:
affinity: ${affinity}
pod: ${pod}
Now we have reusable node-ZONE.yaml
scheduling rules, and if we need to change myserver-pod
, we only have to modify one file — myserver-pod.yaml
. This deduplication might not look like much, but what if your Pod is more complex?
pod:
containers:
- args:
- --ignore-db-dir
- lost+found
cpu:
max: 500m
env:
- MYSQL_ROOT_PASSWORD=yourpassword
expose:
- mysql: 3306
image: mysql
name: mysql
volume:
- mount: /var/lib/mysql
store: mysql-persistent-storage
labels:
name: mysql
name: mysql
version: v1
volumes:
mysql-persistent-storage:
fs: ext4
vol_id: bd82f7e2-wece-4c01-a505-4acf60b07f4a
vol_type: cinder
Add a few sidecar containers, and this manifest is clearly too complex to maintain multiple copies of. Here’s three versions of the mysql
Pod using imports to avoid duplication:
imports:
- affinity: ./node-us-east-1a.yaml
- pod: ./mysql-pod.yaml
params:
affinity: ${affinity}
pod: ${pod}---imports:
- affinity: ./node-us-west-1a.yaml
- pod: ./mysql-pod.yaml
params:
affinity: ${affinity}
pod: ${pod}---imports:
- affinity: ./node-ap-southeast-1a.yaml
- pod: ./mysql-pod.yaml
params:
affinity: ${affinity}
pod: ${pod}
The sharedmysql-pod.yaml
template:
params:
- affinity: the Pod's affinities (for scheduling)
pod:
affinity: ${affinity}
containers:
- args:
- --ignore-db-dir
- lost+found
cpu:
max: 500m
env:
- MYSQL_ROOT_PASSWORD=yourpassword
expose:
- mysql: 3306
image: mysql
name: mysql
volume:
- mount: /var/lib/mysql
store: mysql-persistent-storage
labels:
name: mysql
name: mysql
version: v1
volumes:
mysql-persistent-storage:
fs: ext4
vol_id: bd82f7e2-wece-4c01-a505-4acf60b07f4a
vol_type: cinder
Koki Short’s imports help you manage manifests by removing the need to duplicate code. Write each piece once, and you only have one copy to maintain. In this post, we’ve only shown how to parameterize affinity
, but the same pattern applies to sidecar containers and other fields (and other resources) as well.
Links
Read more about Koki Short and its module system:
docs.koki.io/short/modules/
Affinity-based scheduling:
docs.koki.io/short/resources/pod/#node-affinity
Download the examples from this post (and more):
github.com/koki/compendium
The Koki Short project:
github.com/koki/short
Try out Short syntax live in Chrome with our Koki Kubernetes Viewer extension.