Resize Persistent Disk in Google Container Engine

Currently, there doesn’t seem to be any automated way to support resizing of persistent disk in GKE. In other words you can’t simply run:

$ gcloud compute disks resize my-disk-1 --size=1TB

and have the Kubernetes container pick it up…

I had two different scenarios. One, I defined a partition number in my deployment.yaml because I built the persistent disk from a image with a predefined size and partition and two, I didn’t build from an image and just created the persistent disk with no partition information. Let's talk about the former created like this:

$ gcloud compute images create jenkins-home-image --source-uri \
$ gcloud compute disks create --size=50GB jenkins-home \ 
--image jenkins-home-image

Couple things to note here. This image is 10G. If the image has a predefined size and partition like this one you’re still going to have to fdisk and resize even if you set the size on the persistent disk to be 50G when creating it.

WARNING: Some requests generated warnings:
- Disk size: '50 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See for details.

Here is the yaml:

- gcePersistentDisk:
fsType: ext4
partition: 1
pdName: jenkins-home
name: jenkins-home

So… I had to log into the VM instance where the container was running and actually run fdisk. First you delete the partition then recreate it with the same start sector number and then write it.

You can find the device by running lsbk:

$ lsblk |grep -1 kubernetes
sdb 8:16 0 50G 0 disk
└─sdb1 8:17 0 50G 0 part /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/c195bc3a-b055-11e7-9337-42010a8e001b/volumes/

Do your fdisk stuff as mentioned above and after that run:

$ partx -u /dev/sdb

Once I did this I was able to run resize2fs against the device:

$ resize2fs /dev/sdb1

Prior to running the fdisk, resize2fs would just let you know you’re already using all the blocks. Once all that is said and done you should be able to see the correct size from inside your container.

$ kubectl exec jenkins-3730059860-td55q --namespace jenkins \
-c master -- df -h
Fetching cluster endpoint and auth data.
kubeconfig entry generated for iac-test-cluster.
Filesystem Size Used Avail Use% Mounted on
overlay 95G 3.7G 91G 4% /
tmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/sda1 95G 3.7G 91G 4% /etc/hosts
/dev/sdb1 50G 353M 47G 1% /var/jenkins_home

The latter scenario was pretty easy. I was simply able to log into the VM and run resize2fs. The containers picked it up immediately. These disk were created like so:

$ gcloud compute disks create --size=50GB my-data

The yaml is here:

- gcePersistentDisk:
fsType: ext4
pdName: my-data
name: my-data

So hopefully while we wait for this:

..the above was helpful.