Comparison: Nutanix versus VMware VSAN

Software Defined approach in delivering Data Center resources is on the way from the chasm according to Geoffrey Moore ideology and I would like to say that in 2–3 years will come at the peak of its technology adoption life cycle.

I spent some time to try compare two interesting products of Software Defined Storage (SDS) market — Nutanix 4.1 and VMware Virtual SAN 6.0 and my reflections on this research you can find below. And since I am system engineer I am really not satisfied with usual marketing info and tried to understand the depths of engines of these products, thus this comparison will be a lit bit fanatical, but I presume you can then understand why and where one project is better than another and vice versa.

This comparison I decided to split into a several separate fields to simplify structure of comparing.

Data redundancy techniques. Since both Nutanix 4.1 and VMware VSAN 6.0 use duplications to store copies of the data pieces across different physical devices to achieve data redundancy and in the same time eliminating any central elements, they both require minimum 3 nodes per cluster to tolerate one copy loss. Why? Because both products use some kind of paxos algorithm implementation to achieve strong consensus in storing data across the cluster.

VMware VSAN data placement

VSAN leverage witnesses/votes techniques per object to provide majority quorum during any kind of split brains cases (problem when cluster without central/orchestrator/supervisor element will be spit on two or more independent parts), while Nutanix use modified Cassandra ring platform to store metadata of stored data across the cluster nodes.

Nutanix data placement

Thus, for the current moment both products use 2n+1 rule to calculate required minimum number of failure domains (node/block of nodes/rack of nodes) in the cluster. One difference should be note here is the maximum failure to tolerance (FTT), or in Nutanix world — the replication factor (RF), number. For VSAN we can configure to store 4 copies of data, while in Nutanix this maximum number is 3.

Next interesting difference in how actually systems provide SLA granularity if speaking about FTT/RF and other configuration capabilities. In Nutanix, despite the data granulation is achieved on per 1MB block level and redundant copies of it dynamically spread across quorum nodes, the FTT/RF number turned on at per datastores basis which are presented to hypervisors and on which protected VMs are located. This is mean that to provide to VM’s virtual disk the right SLA it must be placed on the right datastore. In VSAN, since SDS directly implemented into the hypervisor kernel, FTT/RF number directly turned on per virtual disk. Note in contrast, for VSAN the redundant data granulation is achieved on per 255GB block level (actually just logically reserves space on the spindle/device in thin manner). But if we speaking about applying these products to vSphere platform, actually where is FTT/RF number turns on is totally not a headache of your operation team (read OPEX the same here), since the Storage Policy Based Management (SPBM) should be used for both to provide the necessary flexibility.

Another interesting difference is an ability to declare a failure domain. In VSAN 6.0 you can combine several nodes into one failure domain, and data copies will be always placed on different nodes, racks or rooms, depending on what has been declared as a failure domain. In Nutanix the failure domain awareness enabled automatically on only per nodes or per blocks basis (“block” here is a Nutanix unit/module with one, two or four nodes) automatically taking into account the usage of non uniform nodes (for example, difference in space capacity across the nodes). But, if you want draw a map of data placement for your SDS, in Nutanix 4.1 you can not declare a failure domain.

Downtimes. Both systems are projected to provide zero downtime for hosted VMs. During scaling, adding new or decommissioning exists hardware, during hardware maintenance (replacement or microcode upgrades) or in case of crush of failure domain, you may or may not, depending on the applied design, achieve only performance degradation. How significant it can be, again, depends on the applied design. Here is one example of possible design.

Performance flexibility. Despite both products use fast SSD caches for catching write commands and retain as much as possible read commands of hot percentiles, the cornerstone in performance flexibility for storage platforms is always an ability provide variability in the levels of service. Let’s talk about what is usually called a tiering. Since VSAN 6.0 strictly demarcated available storage devices across only cache and capacity layers, it is not possible to allocate different performance types to different storage pools. In other words, it is not possible to declare, for example, low-cost (4TB SATA for our days), cost-performance balanced (1.2TB SAS) and high-performance (SSD) pools in the same storage cluster. But do not jump to sharp conclusions, since here should be considered available possibilities of performance tuning per VMs (actually per vmdk file) basis: strip width used to split vmdk evenly across available spindles (as RAID-0) and using read cache reservations you can guarantee allocation a definite part of cache layer for a specific vmdk.

In contrast Nutanix combines usage of SSD drives for caching and capacity layer. Thus in fact it provides two tiers in capacity layer, SSD and HDD, with automatic continuous data blocks relocations across them, but this tiering is hidden and not configurable. To achieve desired variability in the levels of service in Nutanix you can configure several storage pools in single cluster, but instead of combining different drives in separate pools across nodes, you should use nodes with different type of capacity, mainly from different series. In other words, when storage cluster will consist non uniform nodes.

And since both products to comply with copy replication policy use low latency ethernet, one additional note here is a network saturation control. To prevent when replication/reprotection/rebuilding traffic might impact to another type of traffic (VM networks, management, vMotion, etc.) or vice versa, in VSAN might be used Network I/O Control (NIOC) to prioritize, limit or reserve required bandwidth to Virtual SAN traffic. NIOC based on distributed virtual switch (VDS) which is required and bundled with VSAN despite used vSphere edition. For Nutanix usage of NIOC functionality is recommended and available only for vSphere Enterprise Plus edition.

Optimizations. To achieve significant reduce in capacity space usage, Nutanix provides deduplication and compression. The Elastic Dedupe Engine is a deduplication feature that running across all nodes in cluster. It performs continuous inline scan of all write requests with size equal or greater than 64K before placing them to capacity layer, and makes SHA-1 hashing per 16K chunks of them (dedup granularity is 16K here). For writes with size less than 64K, hashing still perform, but as deferred process after actually data block is placed in capacity player.

Compression more significantly reduces capacity usage, but as results in increasing CPU usage in comparison to CPU built-in SHA calculation which is used in deduplication. It also can be performed as continuous inline process and storing already compressed data or as deferred process with previously stored uncompressed data. As compression library Nutanix uses Google Snappy project.

VSAN 6.0 does not provide deduplication and compression techniques.

Security. With ability to use the data at rest encryption on self-encrypted devices the storage platforms provide required protection for user data in any organization which must comply with claims of FIPS 140–2 publications. VSAN leverages a disk controller capability for this and security key resides in the controller memory to lock or unlock access to drives (local key management). In Nutanix a remote SafeNet KeySecure appliance can be used as a key management server (KMS) for device authentication in the same events of nodes start or reboot.

Freedom in design. In the case designing storage platforms for specific tasks or claims to particular conditions the possibility to use wide range of available components (and their variability) is very important. Since products use different distribution models they show a very big difference in this field. VSAN is distributed as software and range of available components limited only by validated compatibility lists (HCL). Thus depending of required tasks and conditions the architect can build nodes which exclusively and accurately will fit to the desired solution. Nutanix uses a completely different model — an hyper-converged appliances which is in essence a preconfigured hardware-software modules. And as results configuration variability is significantly reduced up to the proposed models.

But in contrast to the hardware range, usage of Nutanix storage is not limited only to vSphere platform, as is true for VSAN, and thus doesn’t ties you to choose of particular hypervisor. With Nutanix modules currently can be used VMware vSphere, Microsoft Hyper-V or KVM hypervisor. BTW, management capabilities for the last are actively integrates directly to Nutanix management system.

And a little bit about maximums. Nutanix does not provide strict scaling limits, but as a best practice recommends do not design cluster sizes greater than 48 nodes or it can be pruned via virtualization platform limits. VSAN 6.0 currently has strict limit to 64 nodes per cluster.

Disaster Recovery. In case of the loss of the entire physical site due to natural disasters, technogenic catastrophes or other force majeure, data protection via replication to other availability zones should be considered and accurately designed. Data protections can be designed either with completely zero RTO and RPO or with strictly defined data loss acceptance and clear predictable recovery time. Speaking about storage platforms nowadays, first (zero RPO) can be achieved via synchronous replication and second via asynchronous, when written data arrives to remote site with a little delay, during which data delta loss is acceptable. Both products can be used in “one to one”, “one to many” and “many to many” sources to target topologies. Nutanix has a built-in asynchronous replication engine for any above topologies and additionally offers the synchronous replication ability between two site at a distance up to 400 km with round trip latency no greater than 5 ms.

In case of using VSAN, no internal engine is used, instead of it VMware offers another product which embedded in vSphere editions — vSphere Replication which can be used with any ESXi datastore including VSAN. vSphere Replication also provides compression to reduce replication bandwidth and has ability to use as a target the VMware vCloud Air cloud. But vSphere Replication 6.0 does not provide any kind of synchronous replications.

Both products allow the configuration of the replication groups on per VMs basis.

In addition to performing replications via vSphere Replication the disaster recovery automation engine VMware Site Recovery Manager (SRM) can be used which allows hold a finger on the big red button to start the recovery process via beforehand tested plans (strict boot order, IP and DNS change, etc.). Full support Nutanix integration with SRM is still expected soon.

follow Igor Nemylostyvyi on Twitter