I3 instances and NVMe: booya!

In a previous post, I used an i2.8xlarge instance (32 vCPUs, 244GB RAM, 8x800GB local SSD storage, 10Gb networking) to build FreeBSD in 17 minutes. Pretty good, but in the never ending quest for performance, I decided to try again with an i3.8xlarge instance, as the I3 family is now supported by the latest FreeBSD 11.1 AMI released at the beginning of August. Kudos to Colin Percival for maintaining the FreeBSD AMIs!

The I3 family

The I3 family is more than a generation bump. Along with the FPGA instances (F1 family), they’re the first to support non-volatile memory express (NVMe) SSD instance store volumes, which deliver a massive performance upgrade compared to traditional SSDs. Thanks to this new technology, the I3 family can deliver up to 3.3 million IOPS at a 4 KB block and up to 16 GB/second of sequential disk throughput.

Along with a CPU upgrade (from Ivy Bridge to Broadwell), we should definitely see a nice improvement in our build time. Let’s get to work.

Launching an I3 instance

We’ll start by launching an i3.16xlarge instance (64 vCPUs, 488GB RAM, 8x1900GB local NVMe storage, 20Gb networking), with the latest FreeBSD 11.1 AMI (ami-ab56bed2 in eu-west-1).

I’m in love already

A few minutes later, we can connect to the instance. Let’s take a look at the NVMe devices.

Preparing the build

Now, let’s create two ZFS pools with four disks each: one for /usr/src, one for /usr/obj.

OK, now let’s fetch and extract the FreeBSD distribution.

The moment of truth

Let’s build. Ready… set… go!

And the result is… 10 minutes and 54 seconds.

This is a 35% speedup compared to the 17 minutes we got on the i2.8xlarge. Granted, the i3.16xlarge has more cores, but given the structure of the FreeBSD build, they’re hardly ever put to work. Amdahl’s Law strikes again.

I’m guessing that most of the improvement does come from the crazy-fast NVMe storage system. To confirm this, let’s run the same test using memory disks.

No improvement at all. This leads me to think that CPU is now the bottleneck. Indeed, the quest never ends. Now, EC2 team, could be please get NVMe on the upcoming C5 Skylake-based instances? Please please please? :D

Thanks for reading and happy testing.