Cisco Nexenta ZFS Storage Appliance Configuration and Benchmarking
The Cisco Unified Computing System is a powerful foundation for the Cisco Storage platform. By using Nexenta software, we have put together many different designs of storage systems, each optimized for different functions. I recently built a ZFS Storage Appliance in the Adcap development lab for validation and testing using the Cisco C240 server as the controller. Over the last week I have also built out a flexible testing system to put the box through its paces.
Cisco has been working with Nexenta and VMware to test and certify Cisco UCS on since early 2012:
- Cisco website lists Nexenta as certified for deployment on UCS Rack Mount servers
- Cisco website PDF article about using Nexenta on Cisco C210
- Nexenta website page for Cisco and VMware solutions
[caption id=”attachment_7616" align=”aligncenter” width=”640"]
Adcap Development Lab in Alpharetta, GA[/caption]
The Cisco C240 is a dual 8-core processor box with the Intel E5–2665 processor and Sandy Bridge chipset. It has 24 slots for memory — this test box has 96 GB memory out of a possible max of 384 GB. There are 24 internal drives connected to a LSI MegaRAID controller. This box has twenty 300GB drives and two146GB drives with 10K RPM speed. It also has two Intel 710 SSD’s, one at 100GB and the other at 200GB size.
[caption id=”attachment_7617" align=”aligncenter” width=”640"]
Cisco C240 Server Front[/caption]
[caption id=”attachment_7618" align=”aligncenter” width=”640"]
Cisco C240 Server Back[/caption]
Connected to the C240 with an external SAS2 controller are two JBOD’s — one with twenty-four 2.5” drive bays, and the other with twenty-four 3.5” drive bays. The 2.5” JBOD has twelve 300GB 10K RPM drives and two Intel 710 SSD’s, each with 100GB of storage. The 3.5” JBOD has six drives, each with 3TB at 7200 RPM. The JBOD’s are connected back to the C240 using a LSI 9207–8e external SAS controller.
[caption id=”attachment_7619" align=”aligncenter” width=”640"]
Supermicro JBOD’s — 24 x 2.5" and 24 x 3.5"[/caption]
I am running NexentaStor 3.1.35 production version with plugins. Nexenta uses the concept of hybrid storage pools to provide fast storage. For the testing, I have set up two main pools. Each pool is shared as both NFS and iSCSI with 128k block sizes:
- Mirrors — 14 mirrors of 300GB drives and one mirror of 146GB drives. I made as many mirrors as possible composed of drives in different chasses. 2 discs are spares.
- 3TB-RAIDZ2 — A pool with five 3TB SAS2 disks and one spare.
[caption id=”attachment_7620" align=”aligncenter” width=”613"]
Nexenta Volumes — Fast Mirror Set and Large RAID-Z2 Set[/caption]
During the testing, I have been using dual SSD cache for Level 2 Adaptive Read Cache (L2ARC or read cache) and single or mirrored SSD for ZFS Intent Log (ZIL or write cache), used for acknowledging synchronous writes quickly. The primary read cache is the 96GB of memory on the C240.
[caption id=”attachment_7692" align=”aligncenter” width=”645"]
SuperMicro JBOD 216E16[/caption]
[caption id=”attachment_7694" align=”aligncenter” width=”642"]
SuperMicro JBOD 846E26[/caption]
The C240 has a number of different PCIe cards in it. I used the Intel X520-SR2 Dual Converged Network Adapter in 10G Ethernet mode, bonded together with LACP on a VLAN. The bonded interfaces are set up for 9000 MTU, also known as Jumbo frames. It was straightforward setting up Jumbo frames on the Nexenta system without going to bash shell, shown here:
[caption id=”attachment_7622" align=”aligncenter” width=”660"]
Setup Nexenta LACP VLAN Interface with MTU 9000[/caption]
The C240 is connected to dual Cisco Nexus switches, one a Cisco 5548, and the other a Cisco 5596. There is a Virtual Port Channel set up to the Intel CNA on the C240. Jumbo frames are enabled on the two Nexus switches at 9216 MTU.
[caption id=”attachment_7623" align=”aligncenter” width=”800"]
Cisco Nexus 5548 and Nexus 5596 with Nexus 2248[/caption]
For testing, I have multiple VMware IOmeter virtual machines running on a Cisco Unified Computing System. This UCS box is set up with dual Fabric Interconnects and four B200 M2 servers, each one running VMware ESXi 5.0. Jumbo frames are set up on the UCS by using a Quality of Service profile at 9216 MTU, and the individual interfaces have MTU of 9000 set in their service profiles. The VMware virtual switch also has Jumbo frames set up on it with MTU of 9000.
[caption id=”attachment_7625" align=”aligncenter” width=”800"]
Cisco UCS System with Four Blades and Two Fabric Interconnects[/caption]
The VMware IOmeter application is the stock version with only minor modifications. I installed twelve virtual machines in order to be able to really hammer the Nexenta box.VMware recommends setting the IOmeter hard drive to four times the size of the memory in the storage box to prevent caching of data, so each IOmeter drive is set to 400GB. I also enabled jumbo frames on the IOmeters. I tested that end-to-end Jumbo frame worked by pinging from a IOMeter virtual machine to the Nexenta box.
[caption id=”attachment_7626" align=”aligncenter” width=”611"]
IOMeter to NexentaStor Ping Test Showing Jumbo Frame Success[/caption]
To get the IOMeter virtual machines to work right, I had to run a performance test first, then accept the Intel agreement on the IOMeter console, and on subsequent tests it would work properly. After setting up the twelve virtual machines, I could do the rest of the testing using the Nexenta and IOMeter web interfaces.
[caption id=”attachment_7627" align=”aligncenter” width=”436"]
VMware VCenter on Cisco UCS Showing IOMeter Virtual Machines[/caption]
IOmeter has the capability to test many different combinations. In thinking about the pros and cons of the different configuration options, these were my expectations going into the testing:
- iSCSI would be faster than NFS because it is block storage.
- Write tests would be fastest when using mirrored ZIL, then slower with one ZIL SSD device, and slowest with none.
- Read tests would be faster with bigger SSD cache.
- Random reads would be slower than sequential reads because of the large DRAM and SSD caches.
- In write testing, the pool with 15 mirrors would be much faster than the pool with a single RAID-Z2 group.
- In sequential read testing, the RAID-Z2 would show similar performance to the Mirror group due to DRAM ARC read caching.
- Max Read IOPS testing would be limited by the Virtual machines.
The first test was all-in to see how high I could get the IOPS to go. The Max IOPS test does 100% read with 0% random, using 0.5K block size:
[caption id=”attachment_7629" align=”aligncenter” width=”923"]
IOMeter Test of Nexenta Box with 12 Virtual Machines — Max IOPS[/caption]
As shown in the picture, when running the Max IOPS program, the box tops out at about 400,000 IOPS for a combination of NFS and iSCSI. It may be that it can go even higher, but looking at the variation of IOPS across each server, it looks like some are higher and some are lower, so this is probably pretty close to the limit. I will do this test again when I add Fibre Channell interfaces and see if it goes higher. Next I backed down to just the virtual machines attached to the Mirror set, which was 4 NFS and 2 iSCSI.
[caption id=”attachment_7630" align=”aligncenter” width=”959"]
IOMeter Test of Nexenta Box with 6 Virtual Machines — Max IOPS[/caption]
The 6 server result was somewhat less at about 300K IOPS. After that I ran a test at 3 NFS connected virtual machines, one for each physical server on the UCS system.
[caption id=”attachment_7631" align=”aligncenter” width=”956"]
IOMeter Test of Nexenta Box with 3 Virtual Machines — Max IOPS[/caption]
Taking out the iSCSI virtual machines really reduced overall performance when running multiple servers. For testing of relative performance differences, I used one server at a time, because when I used multiple servers the data was inconsistent; one server would typically have better performance than the other two.
Next I wanted to determine relative performance differences between NFS and iSCSI. I ran a number of tests using a single server to determine the differences. Overall, iSCSI performance was better than NFS. Most of the tests were run for 15 minutes, or 900 seconds. IOMeter provides beautiful graphs of individual server performance. These two show the relative performance differences of iSCSI and NFS — or in this case, the relative similarities.
iSCSI SQL Test 64K block Size
[caption id=”attachment_7635" align=”aligncenter” width=”968"]
IOMeter VMware Test Single Host iSCSI SQL64 Charts[/caption]
NFS SQL Test 64K block Size
[caption id=”attachment_7638" align=”aligncenter” width=”958"]
IOMeter VMware Test Single Host NFS SQL64 Charts[/caption]
Then I picked one of the tests to do with different SSD caching configurations. The IOMeter SQL 64k test is 66% read with 100% random using 64k block size, and is a good general test.
[caption id=”attachment_7642" align=”aligncenter” width=”1072"]
Adcap C240 ZFS Storage Appliance SSD Testing on Drive Mirrors[/caption]
The results are not really what I expected. I expected the read and write SSD caches make a significant difference across the board. Looking at the difference between a small cache and a large cache, the biggest gains come from having SSD cache in place to reduce NFS write latency (yes I reran the no cache test on iSCSI and NFS multiple times).
Then I repeated the test on the much slower drive pool, the five disc RAID-Z2 pool composed of 3 TB drives. I expected write performance to be significantly slower because of the way writes are spread among pool members. I was not sure what to expect for read performance.
[caption id=”attachment_7654" align=”aligncenter” width=”1073"]
Adcap C240 ZFS Storage Appliance SSD Testing on 3TB RAIDZ2 Datastore[/caption]
The results are not what I expected at all. It appears that as long as there is some caching, NFS performs better than iSCSI. Except when there is no cache at all, then iSCSI performs best. I will run this test again to make sure the results are consistent. In comparing the performance of the 15 Mirrors of 300GB drives against the single RAIDZ2 group of five 3TB drives, the mirrors are much faster, as expected. However, when comparing cached NFS performance of the RAIDZ2 group against uncached NFS performance of the Mirror group, they are at about parity. For NFS, read and write caching allows the use of larger and slower drives without a significant performance penalty.
The next set of tests were made to assist in specifying different configurations for different application loads. Based on the requirements, the ZFS Storage Appliance drive pools, cache, and memory can be configured for optimal performance.
[caption id=”attachment_7643" align=”aligncenter” width=”1072"]
Adcap C240 ZFS Storage Appliance Testing on Drive Mirrors with SSD Caching[/caption]
At initial glance, it looks like the iSCSI performance is better than the NFS. But in the common use tests that IOMeter has set up, including OLTP, SQL Server, and Workstation, the iSCSI and NFS connected servers perform about the same. It will be interesting to run these same tests on Fibre Channel and Fibre Channel over Ethernet to see if there is any performance difference.
This is the first of a number of different test series as the Adcap engineering team puts the Nexenta based Cisco storage system through its paces. Please let me know if you have questions on any of the configuration or testing, and if you would like to see any different types of tests run.
