Tweaking Persistent Disk Performance

IoT exchange creates a product that’s somewhat like a Dating Site for IOT devices. With so many conflicting protocols out there, determining what devices can talk to each other is no easy task. As such, they quickly found that the amount of churn these millions of IOT devices were putting on their relational database was causing significant performance issues. After doing all the fine-tuning they could think about, they finally turned to the real culprit : a badly configured Persistent disk.

Now, if you remember, GCP offers two types of disk options for VM instances : Local Disks, and Persistent Disks. Local disks are physically connected to the VM, and thus offer extremely high IOPs, at the cost of having no persistence once the VM is spun down. Persistent Disks, on the other hand, are not physically attached, and can remain in existence as VMs may spin up or down.

But to figure out what their problem was, we have to take a step back an explain how performance of Persistent Disks works, based on configurations.

Standard PD — Bigger is faster

There are two types of Persistent Disk offerings : Standard hard drives, or Solid State Drives where the primary difference is performance & cost.

For standard PDs, performance scales as a function of the size of the disk, and has nothing to do with the number of vCPUs. (Unless you are using an 1 vCPU config. This is an edge case where your performance will be throttled as a result of network egress limits.)

Running FIO on a bunch of configurations, we’re able to chart the performance for random IOPS as well as sustained read/write throughput:

SSD-PD perf = Size + vCPUs

SSD Persistent disks are a little different than their standard counterparts. While their performance is a function of disk size, it’s also a function of the number of CPUs in the VM.

Or perhaps stated more eliquoutenly by the official documentation:

SSD persistent disk performance scales linearly until it reaches either the limits of the volume or the limits of each Compute Engine instance. Whichever is lower.

Now, this may seem odd that the performance of your disk scales with CPU count, but remember, Persistent Disks aren’t physically attached to your VM; They are independently located. As such IO onto a PD is a network operation, and thus it takes CPU to do IO. Which means that smaller instances run out of CPU to perform disk IO at higher rates.

To get a sense of this, there’s two main charts we need to look at. First, is how the IOPs and throughput scale based on the number of CPUs:

And the second is how the IOPs and MBs scales as a function of disk size.

As such, when tuning the performance of SSD Persistent Disks, you need to both change the machine type of the instance to increase the per-vm limits and resize your persistent disks to increase IOPS and throughput per persistent disk.

The fix is in!

Looking at these tables, you can figure out what size-to-performance trade-offs you need. For example if you’re expecting 18,000 IOPS, You need at least a 16 core CPU, with at least a 384 GB disk size in order to not be throttled. (Note, I use the phrasing “to not be throttled” here since your config may allow that max throughput, but your application might not be delivering enough IOPS to take advantage of it.)

For IoT exchange, this hit home. THey looked at some of our graphs along with their own benchmarking statistics, and realized that a 32 vCPU machine was needed on their compute core in order to properly write the data they needed.