Trying out the new In-Place Pod Resource Resizing!

I’ve been waiting for the ability to do in-place pod resource resizing for several years now (since 2019!), and I’m really excited to see that it’s finally been released as an alpha feature in Kubernetes 1.27. This enables you to automatically resize a pod’s CPU and memory limits and requests without restarting it. I’ve spent a lot of time researching live software updates and realize how challenging anything “live” can be. I’m grateful for all of the work that the contributors put into this to make in-place resizing happen, as it required a lot of changes across multiple Kubernetes components. (I was not involved in this work, but I am very excited to use it!)

The very simplified version is that this feature makes the pod spec resources mutable by allowing Kubernetes to update the underlying c-group allocation in-place. This is particularly useful in the case of scaling pods vertically, such as with Kubernetes’ built-in Vertical Pod Autoscaler (VPA), which allows applications to scale resources up/down within the same pod (rather than scaling with more pods in/out as with traditional horizontal pod scaling).

There are many use cases where vertical scaling is helpful, such as with stateful database workloads where traffic may be bursty but service disruptions are costly. In another example, there was an exciting talk at KubeCon North America 2022 showing how to use this in-place feature with eBPF.

This blog is a quick example of how to try it out, and the new changes you’ll see in the pod spec. There’s a great video in this blog, and I used it to help setup my example. Here I go into a bit more details about how things look during the update and show all of the steps in the process.

I usually use a public cloud version of Kubernetes, but since version 1.27 is not yet available (as of April 2023) in these managed versions, we’ll spin one up locally using minikube. There are many ways to do this; this is just one simple example.

This is released under the feature flag InPlacePodVerticalScaling. Start a minikube cluster up with that flag, and add a node:

minikube start --kubernetes-version=v1.27.0 --feature-gates=InPlacePodVerticalScaling=true --cni calico
minikube node add

When that is ready, start a test pod. Let’s say for example that with our application, it is safe to change the amount of CPUs with no restart, but changing the amount of memory requires a restart. For example, a pod running a database has no problem with a CPU count change while running, but decreasing the amount of memory would cause unexpected behavior. We show this in the pod yaml by setting restartPolicy to be RestartContainer for “memory”. Otherwise the default behavior will be to attempt to update the resource in-place.

apiVersion: v1
kind: Pod
metadata:
  name: inplacedemo
spec:
  containers:
  - name: inplacedemo
    image: alpine
    command: ["tail", "-f", "/dev/null"]
    resizePolicy:
    - resourceName: "memory"
      restartPolicy: "RestartContainer"
    resources:
      limits:
        cpu: 2
        memory: "1Gi"
      requests:
        cpu: 1
        memory: "500Mi"

Apply the yaml and make sure it’s running, then take a look at the new fields by getting the pod info when it’s ready:

$ kubectl apply -f yaml_above.yaml
pod/inplacedemo created
$ kubectl get pods
NAME          READY   STATUS    RESTARTS   AGE
inplacedemo   1/1     Running   0          53s
$ kubectl get pod inplacedemo -oyaml

The first thing to notice under pod spec is the resizePolicy :

spec:
  containers:
    ...
    resizePolicy:
    - resourceName: memory
      restartPolicy: RestartContainer
    - resourceName: cpu
      restartPolicy: NotRequired

You’ll also notice under the “status” field that there are some new fields. allocatedResources are the resources allocated to the pod’s containers, and there is an additional resources field. At first I was confused about why the duplicate resource field, but then I realized that this reflects the actual current resources rather than the desired/to-be-updated ones (more next). Notice this is all under containerStatuses :

 containerStatuses:
  - allocatedResources:
      cpu: "1"
      memory: 500Mi
    ...
    resources:
      limits:
        cpu: "2"
        memory: 1Gi
      requests:
        cpu: "1"
        memory: 500Mi

Let’s start by modifying the pod CPU limits, increasing it from 2 to 3. We’ll do this on the command line via patching:

kubectl patch pod inplacedemo --patch '{"spec":{"containers":[{"name":"inplacedemo", "resources":{"limits":{"cpu":"3"}}}]}}'

If you look at the pod now (kubectl get pod inplacedemo -oyaml), you’ll likely (but not necessarily) see the resize field appear, and you’ll also see the pod spec resources with the new value and the pod status resources with the old value:

spec:
  ...
    resources:
      limits:
        cpu: "3"
        memory: 1Gi
      requests:
        cpu: "1"
        memory: 500Mi

status:
  ...
  containerStatuses:
    ...
    resources:
      limits:
        cpu: "2"
        memory: 1Gi
      requests:
        cpu: "1"
        memory: 500Mi
    restartCount: 0
  ...
  resize: InProgress

Your milage may vary on how long it takes for the pod to go from resize: InProgress to complete (flag disappears). If you see some other flag such as resize: Infeasible , check your node resources to make sure they are sufficient.

Moving on to memory, let’s increase the limits from 1G to 2G:

kubectl patch pod inplacedemo --patch '{"spec":{"containers":[{"name":"inplacedemo", "resources":{"limits":{"memory":"2Gi"}}}]}}'

Now, the process is the same as before with the resize tag and pod status. Make sure the resize has completed by checking that field or the resource status. (In my setup, this takes about 15s-1 minute. Note there is a bug right now where it may take longer.) After that, verify that the restarted based our our flag:

$ kubectl get pods
NAME          READY   STATUS    RESTARTS      AGE
inplacedemo   1/1     Running   1 (88s ago)   25m

And that’s it! I’m really looking forward to trying this out in all of my stateful applications that require vertical pod autoscaling, and I’m looking forward when that feature is integrated into VPA!