Use nconnect to effortlessly increase NFS performance

Having only 1 network connection per NFS mount is so 2019.

Linux has a new feature called “nconnect” that enables multiple TCP connections for a single NFS mount. Setting nconnect as a mount option enables the NFS client to open multiple “transport connections” for the same host.

nconnect is included in linux kernel versions >= 5.3. (So it’s available in Ubuntu 20.04, which uses Linux kernel 5.4).

It’s easy to utilize. Just add an nconnect=_ value to the options list (-o).

mount -t nfs -o ro,nconnect=16 198.18.0.100:/datasets /mnt/datasets

Note: The current limit of client-server connections opened by nconnect is 16.

Now, completely under the hood, your NFS client will open multiple TCP connections. Applications don’t need to change since a single stream of data is still returned to them.

Example

Let’s look at a performance comparison for a single host with an example NFS server (Pure Storage FlashBlade).

Setting “nconnect=16” when mounting the NFS server is an easy way to improve performance for single-client, bandwidth-heavy applications.

In the “Before” test, the host is bottlenecked by the throughput limits that a single TCP connection can provide, i.e. the max bandwidth from a single NIC on the host to a single blade on the storage server. In this example, the limit is 10 Gbps per TCP connection.

Since the storage system includes more than one blade, we should take advantage of its full potential.

In the “After” test, the host can read ~70 Gbps, which is the limit for my example setup that has 7 blades.

Takeaways

Previously, single-client bandwidth-intensive applications were throttled because they had a hard time accessing the full power of their infrastructure. A performant application running on performant compute hardware connected to performant storage was limited by how much traffic it could access over a single TCP connection.

Even some of the most compute-intensive workloads, like deep learning training, are using applications that read data from storage as a single client.

Now, with the nconnect mount option, applications can easily take full advantage of the infrastructure’s capabilities and deliver faster results.

Note: This post is meant as an intro to the nconnect feature — not as a complete guide. Before using nconnect with my example NFS device, FlashBlade, please reach out to support@purestorage.com to learn more about their current best practices.

p.s. Tried to use nconnect and it’s not taking effect?
Try using a new datavip that has no current mounts to your host. A single IP address is represented with one client structure inside of the kernel, so the same nconnect setting will be used across all mounts to that datavip. (Every subsequent mount will use the nconnect settings defined in the first mount to that datavip.)