Scheduling Kubernetes Pods on AWS

Provider-specific details influence the way you manage your Kubernetes applications. For example, AWS EBS volumes can’t be mounted from a different zone. If your pod needs a specific volume, it must be in the same zone as the volume.

AWS availability zones are spread throughout the globe (source)

To improve readability, I’ll use Koki Short, a simpler but equivalent syntax for Kubernetes manifests. Here’s a Pod configured specifically to run in us-east-1a:

pod:
  affinity:
  - node: failure-domain.beta.k8s.io/zone=us-east-1a
  containers:
  - image: me/my-server

It uses Node Affinity (read about it here) to make the Pod run on a Node with the failure-domain.beta.k8s.io/zone label us-east-1a.

If we have other clusters in different regions, we can configure the Pod for us-west-1a or even ap-southeast-1a:

pod:
  affinity:
  - node: failure-domain.beta.k8s.io/zone=us-west-1a
  containers:
  - image: me/my-server
---
pod:
  affinity:
  - node: failure-domain.beta.k8s.io/zone=ap-southeast-1a
  containers:
  - image: me/my-server

It’s great that we can write Pod manifests that run in a specific zone, but we don’t want to maintain duplicated YAML. The manifests above are nearly identical — the only change is from us-east-1a to us-west-1a or ap-southeast-1a.

Let’s factor these Pods into reusable components. For example, we can pull out the Node Affinity rule:

# node-us-west-1a.yaml
affinity:
- node: failure-domain.beta.k8s.io/zone=us-west-1a

And import it:

imports:
- affinity: ./node-us-west-1a.yaml
pod:
  affinity: ${affinity}
  containers:
  - image: me/myserver

We can also pull out the rest of the Pod spec:

# myserver-pod.yaml
params:
- affinity: the Pod's affinities (for scheduling)
pod:
  affinity: ${affinity}
  containers:
  - image: me/myserver

And import it:

imports:
- affinity: ./node-us-west-1a.yaml
- pod: ./myserver-pod.yaml
  params:
    affinity: ${affinity}
pod: ${pod}

Now we have reusable node-ZONE.yaml scheduling rules, and if we need to change myserver-pod, we only have to modify one file — myserver-pod.yaml. This deduplication might not look like much, but what if your Pod is more complex?

pod:
  containers:
  - args:
    - --ignore-db-dir
    - lost+found
    cpu:
      max: 500m
    env:
    - MYSQL_ROOT_PASSWORD=yourpassword
    expose:
    - mysql: 3306
    image: mysql
    name: mysql
    volume:
    - mount: /var/lib/mysql
      store: mysql-persistent-storage
  labels:
    name: mysql
  name: mysql
  version: v1
  volumes:
    mysql-persistent-storage:
      fs: ext4
      vol_id: bd82f7e2-wece-4c01-a505-4acf60b07f4a
      vol_type: cinder

Add a few sidecar containers, and this manifest is clearly too complex to maintain multiple copies of. Here’s three versions of the mysql Pod using imports to avoid duplication:

imports:
- affinity: ./node-us-east-1a.yaml
- pod: ./mysql-pod.yaml
  params:
    affinity: ${affinity}
pod: ${pod}
---
imports:
- affinity: ./node-us-west-1a.yaml
- pod: ./mysql-pod.yaml
  params:
    affinity: ${affinity}
pod: ${pod}
---
imports:
- affinity: ./node-ap-southeast-1a.yaml
- pod: ./mysql-pod.yaml
  params:
    affinity: ${affinity}
pod: ${pod}

The sharedmysql-pod.yaml template:

params:
- affinity: the Pod's affinities (for scheduling)
pod:
  affinity: ${affinity}
  containers:
  - args:
    - --ignore-db-dir
    - lost+found
    cpu:
      max: 500m
    env:
    - MYSQL_ROOT_PASSWORD=yourpassword
    expose:
    - mysql: 3306
    image: mysql
    name: mysql
    volume:
    - mount: /var/lib/mysql
      store: mysql-persistent-storage
  labels:
    name: mysql
  name: mysql
  version: v1
  volumes:
    mysql-persistent-storage:
      fs: ext4
      vol_id: bd82f7e2-wece-4c01-a505-4acf60b07f4a
      vol_type: cinder

Koki Short’s imports help you manage manifests by removing the need to duplicate code. Write each piece once, and you only have one copy to maintain. In this post, we’ve only shown how to parameterize affinity, but the same pattern applies to sidecar containers and other fields (and other resources) as well.

Links

Read more about Koki Short and its module system:
docs.koki.io/short/modules/

Affinity-based scheduling:
docs.koki.io/short/resources/pod/#node-affinity

Download the examples from this post (and more):
github.com/koki/compendium

The Koki Short project:
github.com/koki/short

Try out Short syntax live in Chrome with our Koki Kubernetes Viewer extension.