StarWind Virtual SAN (VSAN) vs Microsoft Storage Spaces Direct (S2D), Part 1: Hyper-V HCI Performance Benchmarking (TCP)

Introduction

I/O is what either makes or breaks the system. Choosing the right data storage solution is crucial for your system’s performance and reliability. If you’re a Hyper-V fanboy and haven’t spent the last ten years living under a rock, you’re likely aware of both StarWind Virtual SAN (VSAN) and Microsoft Storage Spaces Direct (S2D). So, which one should you go for? In this article, we dive into these two favorable options, comparing their performance, storage efficiency, and how they work in a 2-node Hyper-V cluster setup. By the end, you’ll have a better idea of which solution could be the best fit for you.

To compare these two fairly, we set up a 2-node Hyperconverged Infrastructure (HCI) Hyper-V cluster under two different configurations:

• StarWind VSAN, NVMe-oF, TCP
  • Full host mirroring, basically a ‘network RAID1’, combined with a RAID5 for local NVMe pool protection, so RAID51 for the whole config eventually.
• Microsoft S2D (‘Mirror-accelerated parity’), TCP
  • We tested two specific corner cases: a) the entire test workload is placed in the mirror tier to check the ‘best case’ scenario, and b) the workload is split between the mirror and parity tiers to test ‘real-world’ scenario. We specifically avoided c) which is ‘parity only’ and ‘worst case’ for obvious reasons, it’s just unusable…

Disclaimer: For simplicity, and because it isn’t a production environment, we used replicated disk device as a witness. While this setup works fine in a lab, it should be avoided in production, and a physical, out-of-cluster witness is recommended instead.

High-scope interconnect diagram: StarWind VSAN, NVMe-oF, TCP

In this setup, each Hyper-V node is equipped with 5 NVMe drives, passed through to the StarWind Linux-based Controller Virtual Machine (CVM) using the Hyper-V ‘PCI pass-thru’ mechanism. Inside the CVM, these NVMe drives are assembled into a single RAID5 virtual LUN. On top of this LUN, two StarWind High Availability (HA) devices are created, ensuring data replication and continuous availability. StarWind NVMe-oF Initiator was chosen for uplink connectivity due to the lack of a native Microsoft NVMe-oF initiator (Microsoft was expected to introduce full NVMe-oF support in Windows Server 2025, but this hasn’t happened yet…), and it ‘brings’ these StarWind HA devices to the Hyper-V nodes. Two Cluster Shared Volumes (CSVs) are then created on these connected devices, one per Hyper-V node, according to Microsoft’s best practices.

High-scope interconnect diagram: Microsoft S2D (‘Mirror-accelerated parity’), TCP

This scenario covers two use cases, both within the ‘mirror-accelerated parity’ S2D configuration:

1. Workload placed in the mirror tier to maximize performance by keeping data in the faster mirror tier. This is the ‘best case’, achievable only with a very lightly used (or underutilized) S2D cluster. When testing your own S2D clusters with ‘mirror-accelerated parity,’ make sure this isn’t the only case you test, or you’ll end up in trouble!
2. Workload placed across both tiers, simulating a more balanced use case where data moves between the mirror and parity tiers. This reflects more ‘real-world’ conditions, where the workload doesn’t fit entirely in the mirror tier, and the Resilient File System (ReFS) starts offloading data to the parity tier. For the sake of experiment, we also attempted to force a test where writes go directly to the parity tier, the ‘worst case’.

Two storage tiers are set up with different resiliency settings: ‘Mirror’ for ‘performance’ and ‘Parity’ for ‘capacity,’ with the following parameters:

Volumes are allocated with 20% in the mirror tier and 80% in the parity tier, adhering to Microsoft’s recommendations:

ReFS manages the data movement between these tiers to optimize performance. The threshold value, at which ReFS starts moving data between the tiers, was left at the default – 85%.

Capacity efficiency

Storage efficiency is a big deal in storage solutions!

• StarWind VSAN
  In this configuration (5x NVMe drives per Hyper-V cluster node), it achieves 40% storage efficiency since VSAN combines host mirroring with RAID5 for local NVMe pool protection.
• Microsoft S2D (‘Mirror-accelerated parity’)
  In this setup, it delivers 35.7% storage efficiency (20% mirror, 80% parity), though this can vary based on the volume percentage allocated to the mirror tier. For more on calculating storage efficiency for mirror-accelerated parity, check out the provided link.

Microsoft recommends leaving some space unallocated in the storage pool to allow volumes to perform ‘in-place’ repairs after an individual drive failure. If enough free space is available, an immediate, in-place, parallel repair can restore volumes to full resiliency even before the failed drive is replaced. This process happens automatically. In our setup, the recommended reserved space is 5.82 TB (20% of the total pool size). See:

When planning your own S2D deployment, keep these factors in mind to make sure you get the best performance and storage efficiency for your needs!

Testbed Overview:

When evaluating StarWind VSAN and Microsoft S2D storage solutions, we didn’t cut any corners. Our testbed was solid and carefully configured to mirror real-world environments, ensuring our results are relevant, reliable, and reproducible. Here’s a snapshot of the hardware and software components that powered our tests:

Hardware:

Software:

StarWind CVM parameters:

Testing methodology:

Benchmarks were run using the FIO utility in client/server mode. We set up a total of 20 virtual machines (VMs), with 10 VMs on each server node. Each VM had 4 vCPUs, 8GB of RAM, and three RAW virtual disks connected to separate virtual SCSI controllers.

Test Scenarios:

• Microsoft Storage Spaces Direct (‘Mirror-accelerated parity’)
  • Mirror-only. For tests where the workload is placed entirely in the mirror tier, each virtual disk size was limited to 10GB in size.
  • Mirror+Parity. For tests utilizing both the mirror and parity tiers, each virtual disk size was set to 100GB.
• StarWind Virtual SAN
  • For all tests, each virtual disk was 100GB in size.

Data Patterns Tested:

• 4k random read
• 4k random read/write (70/30)
• 4k random write
• 64k random read
• 64k random write
• 1M read
• 1M write

Pre-Test Warm-Up:

Before running tests, we filled the virtual disks with random data and ‘warmed them up’ with specific patterns to ensure stable flash performance.

• 4k random read/write (70/30) and 4k random write: VM disks were warmed up with a 4k random write pattern for 4 hours.
• 64k random write: VM disks were warmed up with a 64k random write pattern for 2 hours.

Test Execution:

• Duration: Read tests were conducted for 600 seconds, and write tests lasted 1800 seconds.
• Repetition: All tests were repeated three times, and the average value was used as the final result.

Specific Configurations:

• Microsoft Storage Spaces Direct (S2D)
  Following Microsoft’s recommendations for the S2D+ReFS scenario, test VMs were placed on the CSV owner node to avoid ReFS redirecting requests to another node, ensuring local data reads without using the network stack and providing less network utilization on writes. Each VHDX file was placed in different subdirectories to optimize ReFS metadata operations and reduce latency.
• StarWind Virtual SAN (VSAN)
  VMs were evenly distributed across hosts without being pinned to the node that owns the volume. Each VHDX file was placed in different subdirectories to maintain consistent performance.

Benchmarking local NVMe performance:

Before diving into the full evaluation, we checked if the NVMe drives lived up to the vendor’s promises by running a series of tests to see if their performance matched up. Here’s an image showing the vendor-claimed performance:

Using the FIO utility in client/server mode, we tested the performance of the NVMe SSDs in our server within a local storage setup. We applied different patterns to see how the NVMe SSDs handled various types of data. The results are shown below:

Our tests showed that the NVMe drives lived up to what the vendor promised. Whether handling small 4k reads or large 1M writes, they delivered on speed and consistency.

Benchmark results in a table:

The benchmarking results are presented in tables to illustrate performance metrics such as IOPS, throughput (MiB/s), latency (ms), and CPU usage. An additional metric, “IOPS per 1% CPU usage,” highlights the performance dependency on the CPU usage for 4k random read/write patterns. This parameter is calculated using the following formula:

IOPS per 1% CPU usage = IOPS / Node count / Node CPU usage

Where:

• IOPS represents the number of I/O operations per second for each pattern.
• Node count is 2 nodes in our case.
• Node CPU usage denotes the CPU usage of one node during the test.

By incorporating this additional metric, we aimed to provide deeper insights into how CPU usage correlates with IOPS, offering a more nuanced understanding of performance characteristics.

Now let’s delve into the detailed benchmark results for each storage configuration.

StarWind Virtual SAN (VSAN)

The table illustrates StarWind VSAN’s performance under various workload patterns and configurations. For 4k random reads, IOPS scored from 420,000 at lower queue depths to 881,000 at higher depths. In a mixed 4k random read/write (70/30) test, it achieves up to 561,000 IOPS, showcasing its prowess in handling mixed workloads.

In the 64k and 1M read/write patterns, the StarWind VSAN reaches up to 15,2 GB/s, demonstrating its ability to handle these workloads effectively.

Overall, StarWind VSAN shows great performance at 4k random read/write patterns, consistent read and write performance regardless of VM location, and impressive storage efficiency at 40%.

Microsoft Storage Spaces Direct (‘Mirror-accelerated parity’): Mirror-only

The table below shows S2D’s performance with a ‘mirror-accelerated parity’ configuration, focusing on workloads that 100% fit into the mirror tier.

For 4k random read scenarios, IOPS peak at 2,653,000, highlighting exceptional read performance thanks to 100% local-bound reads. In the 4k random read/write (70/30) pattern, results reach up to 654,000 IOPS.

The 64k random read/write and 1M read/write tests maintain high throughput, with 53,5 GB/s for 64k reads and 52,4GB/s for 1M reads. S2D shows exceptional read performance when VMs are on the volume-owning node, and robust write performance within the mirror tier. However, read performance declines if local reading requirements aren’t met and unusual performance drops occur at certain queue depths. Additionally, write performance can drop if VMs are running on a node that is not the volume owner.

Microsoft Storage Spaces Direct (‘Mirror-accelerated parity’): Mirror+Parity

The performance metrics for the dual-tier configuration in S2D highlight workload placed across both mirror and parity tiers.

In 4k random read patterns, IOPS reach up to 2,500,000, showcasing excellent scalability. The 4k random read/write (70/30) pattern results show up to 247,000 IOPS.

For 64k random read/write and 1M read/write tests, the system maintains strong throughput, with 52,000 MiB/s for 64k reads and 50,100 MiB/s for 1M reads, demonstrating S2D’s robust capability to handle complex data operations across tiers. However, its write performance drops when workloads exceed the mirror tier, which is expected.

Overall, S2D shows exceptional performance in both test cases.

Storage efficiency is about 35.7% and could be even less if additional space is assigned for in-place repairs.

Benchmarking results in graphs:

With all benchmarks completed and data collected, we can now compare the results using graphical charts for a clearer understanding.

4k random read:

Let’s start with the 4K random read test, where Figure 1 showcases the performance in IOPS. The S2D in the ‘Mirror-accelerated parity’ configuration with 100% of the workload in the mirror tier reaches a remarkable 833,000 IOPS at 4 I/O queue depth, scaling up to 2,653,000 IOPS at 128 I/O queue depth.

Comparatively, the StarWind VSAN peaks at 881,000 IOPS at 128 I/O queue depth. Here, S2D outshines StarWind VSAN with approximately 200% more IOPS at higher depths.

So, what’s the magic behind S2D’s performance? It’s all about local reading. In a cluster shared volume (CSV) setup, S2D leverages the SMB 3.0 protocol to allow multiple hosts to access and perform I/O operations on a shared volume (if you want to explore this topic in more detail, please read here or check this article). If a VM is running on the node that owns the volume, it can read data directly from the local disk, bypassing the network stack. This local read path minimizes latency and maximizes performance, leading to impressive IOPS numbers.

However, there’s a catch. This local reading perk only works if the VM is on the volume-owning node. If not, the read operations have to go through the network to the owning node, which can and will slow things down! To keep things running smoothly, you need to keep an eye on where your VMs are running and move them to the appropriate nodes as necessary. It’s tricky!

When it comes to random read latency, Figure 2 reveals that Storage Spaces Direct with 100% of the workload within the mirror tier also excels, starting at a low 0.286 ms at 4 I/O queue depth and increasing to 2.897 ms at 128 I/O queue depth. StarWind VSAN starts at 0.570 ms, reaching 8.827 ms at the same depth.

Even with the workload split between the mirror and parity tiers, S2D maintains superior read latency, starting at 0.293 ms and peaking at 3.068 ms. The latency advantage of S2D is again attributed to local reads.

Switching gears to efficiency, Figure 3 compares IOPS per 1% CPU usage during a 4k random read test. Storage Spaces Direct with 100% of the workload in mirror tier proves highly efficient, delivering up to 21,704 IOPS per 1% CPU usage at 64 I/O queue depth, whereas StarWind VSAN peaks at 7,342 IOPS per 1% CPU usage at 128 I/O queue depth. This makes S2D approximately 196% more efficient.

Even when the workload spans both S2D tiers, mirror and parity, it maintains a strong efficiency advantage, reaching 19,909 IOPS per 1% CPU usage at 64 I/O queue depth.

4k random read/write 70/30:

Next, let’s dive into mixed 70/30 read-write patterns. Figure 4 is key for understanding real-world performance because pure read or write workloads are rare in actual production.

Figure 4 shows the number of IOPS during the mixed 70%/30% 4k random read/write tests with Numjobs = 3.

Interestingly, with Storage Spaces Direct, there’s a noticeable drop in performance at queue depths 4 and 8. This performance drop is not observed in StarWind VSAN tests. StarWind maintains consistent performance, hitting 334,000 IOPS at queue depth 4 and 301,200 IOPS at queue depth 8.

In contrast, with 100% of the workload in the mirror tier, S2D’s performance drops to 114,600 IOPS at queue depth 4 and 62,800 IOPS at queue depth 8, representing reductions of approximately 65.7% and 79.1%, respectively, compared to StarWind VSAN. Fortunately, S2D shows a significant rebound in performance starting at QD=16, ultimately scoring about 25% higher than StarWind VSAN under the same conditions.

However, when the workload is distributed across both mirror and parity tiers, S2D struggles due to ReFS continuously moving new data from the mirror tier to the parity tier, which negatively affects performance. As a result, S2D records 126,200 IOPS at queue depth 2, drops to a low of 49,800 IOPS at QD=8, and then peaks at 247,000 IOPS at queue depth 32.

Meanwhile, StarWind VSAN outperforms S2D, achieving 241,500 IOPS at queue depth 2 and 534,000 IOPS at queue depth 32, highlighting its superior performance with IOPS figures that are 91.4% higher at queue depth 2 and 116.2% higher at queue depth 32 compared to S2D in the “Mirror+Parity” scenario.

Figure 5 examines latency for the mixed 4K random 70/30 workload. Storage Spaces Direct with 100% of the workload in the mirror tier starts at 0.382 ms at 2 I/O queue depth, reaching 6.899 ms at 64 I/O queue depth. StarWind VSAN, on the other hand, starts at 0.582 ms and goes up to 7.768 ms.

Figure 6 explores the number of IOPS relative to 1% CPU utilization during the same mixed workload.

Storage Spaces Direct with workload within the mirror tier provides up to 10,180 IOPS per 1% CPU usage at 16 I/O queue depth, while StarWind VSAN peaks at 5,394 IOPS per 1% CPU usage. This makes S2D about 89% more efficient.

When workload is touching both tiers, mirror and parity, S2D achieves a maximum of 3,506 IOPS per 1% CPU usage, demonstrating 69.7% less performance compared to the StarWind VSAN at 64 I/Os queue depth.

4k random write:

The ability to maintain consistent write performance across various queue depths is crucial for demanding virtualization environments. Figure 7 shows the amount of IOPS during 4k random write operations.

StarWind VSAN stands out with consistent performance across most queue depths, significantly outperforming S2D (100% of the workload in the mirror tier) from I/O queue depth 2 to 16.

With S2D and 100% of the workload in the mirror tier, there’s a big drop in performance at queue depths 4 and 8. At queue depth 4, Storage Spaces Direct score about 46,800 IOPS which is more than 4 times lower than StarWind VSAN’s 192,000 IOPS figure. At queue depth 8, the gap widens even more and StarWind VSAN ends up being 584% more effective. This result is unexpected, as we anticipated better performance from S2D in this test compared to when the workload spans both mirror and parity tiers.

Interestingly, at queue depth 32 StarWind VSAN loses the advantage scoring 167,000 IOPS, while S2D with 100% of the data in the mirror tier gets traction achieving 186,000 IOPS. That’s being said, when workload is hitting both tiers, S2D is unable to show better performance figures and ends up scoring 74,900 IOPS.

Latency for 4K random writes is also a critical factor. Since latency corresponds to prior IOPS results, the overall picture remains consistent in Figure 8.

StarWind VSAN demonstrates the lowest latency, starting at 0.859 ms with a 2 I/O queue depth and increasing to 3.689 ms at a 16 I/O queue depth. Virtual SAN significantly outperforms Storage Spaces Direct with 100% of the workload in the mirror tier, which starts at 1.499 ms (43% higher latency) and rises to 14.876 ms (75% higher latency) in QD=16 test.

When comparing StarWind to S2D with workload within both tiers, mirror and parity, the performance gap is even more noticable, with StarWind showing 63% lower latency at 2 I/O queue depth (0.859 ms vs. 2.324 ms) and 55% lower latency at 32 I/O queue depth (11.476 ms vs. 25.659 ms). Only at 32 I/O queue depth, StarWind VSAN demonstrates slightly higher latency, reaching 11.476 ms compared to S2D’s 10.174 ms, which is about 11% lower.

Efficiency in 4K random write workloads is measured in IOPS per 1% CPU usage, as shown in Figure 9.

Storage Spaces Direct with 100% of the workload in the mirror tier achieves up to 5,167 IOPS per 1% CPU usage, while StarWind VSAN peaks at 3,093 IOPS per 1% CPU usage, making S2D approximately 67% more efficient. However, when the workload utilizes both S2D tiers, mirror and parity, efficiency drops significantly, with a maximum of only 1,291 IOPS per 1% CPU usage, making it the least efficient of the three scenarios.

64k random read:

Moving to larger data blocks, Figure 10 illustrates the throughput performance for 64K random reads.

Storage Spaces Direct with 100% of the workload in the mirror tier significantly outpaces StarWind VSAN, achieving a peak of 53,500 MiB/s at 32 I/O queue depth compared to StarWind’s 14,562 MiB/s. This indicates that S2D delivers approximately 267% more throughput.

When the workload utilizes both S2D tiers, mirror and parity, it shows slightly lower throughput but still surpasses StarWind VSAN significantly. The higher performance in S2D is attributed to local reads, but remember, this efficiency is conditional on the VM running on the node that owns the volume. StarWind VSAN, in contrast, provides stable performance regardless of VM placement, eliminating the need for additional monitoring and VM-to-host binding.

Figure 11 shows the latency for 64K random reads. The results align with the throughput data discussed earlier.

Here, S2D with 100% of the workload in the mirror tier maintains low latency due to local reads, starting at 0.376 ms and reaching 2.243 ms at 32 I/O queue depth. The StarWind VSAN starts higher at 0.749 ms and peaks at 8.343 ms, which is up to 73% higher latency than S2D.

In Figure 12, we examine CPU usage during 64K random reads.

Storage Spaces Direct with 100% of the workload in the mirror tier starts at 17% CPU usage at 2 I/O queue depth and peaks at 38% at 32 I/O queue depth. When the workload hits both tiers, mirror and parity, S2D shows consistent CPU usage trends, closely following the S2D “mirror-only” test results.

StarWind VSAN begins significantly higher at 35% and peaks at 41%, slightly above S2D. This indicates that S2D is more efficient in CPU usage, with StarWind VSAN using approximately 106% more CPU at I/O queue depths 2 to 16 and about 8% more at I/O queue depth of 32.

64k random write:

Figure 13 illustrates the 64K random write throughput, highlighting performance differences across three scenarios.

Storage Spaces Direct with 100% of the workload in the mirror tier exhibits erratic performance, with notable drops at medium I/O queue depths. For example, throughput falls to 2,019 MiB/s at a 4 I/O queue depth, dips further to 1,457 MiB/s at 8 I/O queue depth, and then rebounds to 2,616 MiB/s at 16 I/O queue depth. This pattern reflects the behavior observed in 4K random write tests. Not stable. No good!

In contrast, StarWind VSAN delivers more consistent performance, surpassing S2D by 79.6% at a 4 I/O queue depth, by 167.6% at 8 I/O queue depth, and by 52% at 16 I/O queue depth.

When workloads span both tiers, mirror and parity, S2D shows significantly lower throughput across the board. StarWind VSAN outperforms S2D by 199% at a 1 I/O queue depth, with the performance gap widening to 343% at a 32 I/O queue depth.

This highlights StarWind’s capability to handle write operations with consistently high performance across varying I/O queue depths.

Figure 14 displays the latency for 64K random writes, showing a similar trend.

StarWind VSAN delivers faster response times at lower I/O queue depths (4, 8, and 16), but S2D (with 100% of the workload in the mirror tier) takes the lead at a 32 I/O queue depth, achieving a lower latency of 22.138 ms compared to StarWind’s 30.150 ms.

When compared to S2D’s configuration with workloads spread across both tiers, mirror and parity, StarWind VSAN is significantly more efficient, providing 66.9% faster response times at a 1 I/O queue depth and 77.3% lower latency at a 32 I/O queue depth.

Figure 15 highlights CPU usage during 64K random writes.

StarWind VSAN consistently shows higher CPU utilization compared to both Storage Spaces Direct configurations (Mirror-only and Mirror+Parity). At a 1 I/O queue depth, StarWind uses 25% CPU, which is 79% higher than S2D’s 14% with 100% of the workload in the mirror tier and 92% higher than S2D’s 13% in the mixed-tier test. This trend persists across different I/O queue depths, with StarWind maintaining higher CPU usage but delivering more consistent performance under varying workloads.

1M read:

Figure 16 presents the throughput results for 1024K reads, where S2D with 100% of the workload in the mirror tier significantly outperforms StarWind VSAN, reaching 52,000 MiB/s at a 16 I/O queue depth compared to StarWind’s 15,600 MiB/s — about 233% higher throughput.

Even when workloads are spread across both tiers, mirror and parity, S2D continues to outperform StarWind VSAN by a substantial margin. This impressive read performance from S2D is again due to local reads when the test VM is located on the CSV owner node.

Figure 17 shows the latency results during the 1024K read test, reflecting a pattern similar to the throughput results.

S2D with 100% of the workload in the mirror tier demonstrates impressively low latency, benefiting from local reads. Latency starts at 1.002 ms and increases to 6.098 ms as I/O queue depth grows.

In contrast, StarWind VSAN starts at 1.998 ms and peaks at 20.625 ms, resulting in S2D delivering up to 240% lower latency than StarWind.

When workloads are distributed across both S2D tiers, mirror and parity, latency remains nearly identical to that of the mirror tier tests. It’s all thanks to the local reads!

Figure 18 highlights CPU usage during 1024K reads, where S2D demonstrates significantly lower resource consumption compared to StarWind VSAN.

With workloads in the mirror tier, S2D starts at 5% CPU usage at a 1 I/O queue depth and increases to 16% at a 16 I/O queue depth.

In contrast, StarWind VSAN begins at 26% and rises to 33%, meaning S2D uses about 63% less CPU on average.

Even when workloads span both tiers, mirror and parity, S2D maintains the same CPU usage levels as in the mirror-only benchmarks. S2D’s efficiency in local reads translates to more effective CPU usage, while StarWind requires more resources to sustain consistent performance.

1M write:

When we shift our focus to 1024K sequential write throughput, Figure 19 reveals that Storage Spaces Direct (S2D) with 100% of the workload in the mirror tier holds a significant performance advantage over StarWind VSAN. Specifically, S2D reaches 8,693 MiB/s at a 2 I/O queue depth, while StarWind VSAN manages 3,903 MiB/s. At an 8 I/O queue depth, S2D continues to dominate, hitting 8,559 MiB/s, compared to StarWind’s 4,156 MiB/s. This means S2D delivers approximately 122% higher throughput, if your workload fits into the mirror tier…

However, it’s important to note that this advantage is conditional. If the workload doesn’t fit into the mirror tier, S2D’s performance drops dramatically. This is evident in the multi-tiered test results, where StarWind VSAN outperforms Storage Spaces Direct by about 106% on average.

Diving into the 1024K write latency as depicted in Figure 20, we see a consistent theme.

With 100% of its workload in the mirror tier, Storage Spaces Direct begins at a brisk 2.400 ms and climbs to 18.684 ms at 8 I/O queue depth. In comparison, StarWind VSAN starts at a slower 5.804 ms and escalates to a higher peak of 38.492 ms, which demonstrates that S2D provides up to 106% lower latency.

However, when the workload spans both the mirror and parity tiers, everything gets flipped upside down. S2D records the highest latencies, starting at 13.496 ms and surging to 73.178 ms at 8 I/O queue depth. This again indicates a significant performance shift depending on how well the workload is aligned with S2D’s optimal tier configuration.

Figure 21 highlights CPU usage during 1024K writes. With 100% of the workload in the mirror tier, Storage Spaces Direct (S2D) begins at 8% CPU usage at a queue depth of 1 I/O and consistently holds at 9% across queue depths of 2, 4, and 8 I/O.

In contrast, StarWind VSAN begins at a much higher 24% and remains steady around 25% across all I/O queue depths. This indicates that S2D consumes approximately 72% less CPU on average, demonstrating significantly more efficient resource utilization compared to StarWind VSAN.

When the workload spans both tiers of S2D, mirror and parity, it continues to exhibit even lower CPU usage, starting at just 4% at 1 and 2 I/O queue depths and modestly rising to 5% at 4 and 8 I/O queue depths.

Additional benchmarking: 1 VM, 1 numjobs, 1 iodepth

To gain a deeper understanding of how StarWind VSAN and Storage Spaces Direct perform under specific synthetic conditions, we conducted additional benchmarks focusing on a single VM scenario, with numjobs = 1 and an I/O queue depth of 1. Typically, this is the best way to measure storage access latency.

Benchmark results in a table:

Benchmark results in graphs:

This section presents visual comparisons of the performance and latency metrics across storage configurations under research.

4k random read:

Figure 1 demonstrates IOPS for the 4K random read test at 1 I/O queue depth and with one numjobs. S2D with 100% of the workload in mirror tier outperforms the StarWind VSAN delivering 7,221 IOPS.

This remarkable 550% increase over StarWind’s 1,112 IOPS is primarily due to S2D’s ability to leverage local reads and operate entirely at the host level. In contrast, StarWind VSAN, running inside a VM and mixing local and network I/O, usually faces a much longer data path, which impacts its performance negatively.

Even when S2D operates with data across both mirror and parity tiers, it maintains strong performance at 5,920 IOPS, still surpassing StarWind by 432%.

Latency metrics for the 4K random read test at 1 I/O queue depth, as shown in Figure 2, similarly favor Storage Spaces Direct with 100% of the workload in the mirror tier, which records a swift 0.137 ms. This is a substantial 553% lower latency compared to StarWind VSAN’s 0.897 ms. This advantage is again due to S2D’s local read capabilities and direct host-level operation. Even in a mixed-tier setup, mirror and parity, S2D maintains its lead with a latency of 0.167 ms, still outperforming StarWind by 437%.

Figure 3 showcases the results of the 4K random write test at I/O queue depth=1 with a numjob=1. Storage Spaces Direct with 100% of the workload in the mirror tier achieves a remarkable 5,456 IOPS, an astounding 990% higher than StarWind VSAN’s 501 IOPS. This significant advantage stems from S2D’s ability to write directly to the mirror tier, bypassing the resource-intensive parity calculation and avoiding read-modify-write.

However, when S2D handles workloads across both mirror and parity tiers, performance drops to 2,517 IOPS due to the additional overhead of invalidating data in the parity tier. For a deeper dive into how reading and writing function in a mirror-accelerated parity scenario, please refer to the detailed explanation provided here.

On the other hand, StarWind VSAN, which writes directly to the RAID5 virtual LUN, experiences performance degradation due to the read-modify-write (RMW) operations and the extended I/O data path inherent in its VM-based operation. Despite these technical challenges, the performance of StarWind VSAN at queue depth of 1 appears unusually low, prompting us to initiate an investigation into this issue to uncover the underlying cause.

Moving on to Figure 4, we examine the latency metrics for 4K random writes.

No surprises here. Storage Spaces Direct (S2D) continues to deliver superior performance, benefiting from its efficient data handling within the mirror tier and achieving an impressively low latency of 0.182 ms. This represents a staggering 995% improvement over StarWind VSAN’s 1.991 ms.

Even when S2D operates across both mirror and parity tiers, it maintains a competitive latency of 0.395 ms, still outperforming StarWind by 404%.

In our synchronous 4K read-write single-threaded I/O tests, as shown in Figure 5, Storage Spaces Direct with the dataset entirely in the mirror tier once again takes the lead, achieving 2,887 IOPS — a staggering 1,177% increase over StarWind VSAN’s 226 IOPS.

This significant performance boost is attributed to the same factors observed in asynchronous 4K random write test, where S2D benefits from direct writes to the mirror tier, effectively bypassing the resource-heavy parity calculations and avoiding read-modify-write.

Even in the “mixed tiers” setup, S2D maintains a strong advantage, delivering 1,772 IOPS and still outpacing StarWind by 684%.

Figure 6 highlights the latency results for synchronous 4K read-write single-threaded I/O, further confirming S2D’s performance edge.

With 100% of the workload within the mirror tear, Storage Spaces Direct achieves the write latency of 0.344 ms — an impressive 1,184% lower than StarWind VSAN’s 4.415 ms.

Even when using the mirror and parity tiers, S2D maintains a strong latency advantage at 0.562 ms, outpacing StarWind by 686%. This superior performance stems from S2D’s efficient data handling, consistently delivering lower latency across varying configurations.

Conclusion

To sum it up, both Storage Spaces Direct and StarWind VSAN come with their own set of perks and trade-offs for your IT infrastructure.

Storage Spaces Direct shines in read performance, particularly when virtual machines are aligned with CSV owner nodes. However, we observed some unexpected performance issues during 4K and 64K random-write tests, where S2D sometimes underperformed with data in the mirror tier compared to when it spanned both mirror and parity tiers. This highlights the need for careful monitoring and VM data placing to ensure optimal performance. Mismanagement of workloads can lead to significant performance drops, particularly at certain queue depths. Additionally, S2D requires extra space for fault tolerance, which can impact overall storage efficiency.

On the other hand, StarWind VSAN proves to be a solid choice for high-performance environments, especially with mixed read/write or write-heavy workloads. It consistently delivers superior write performance under load, regardless of VM placement, and offers better capacity efficiency. However, StarWind VSAN lacks the local read boost that S2D provides, can be more demanding on CPU resources, and showed some anomalies in single-threaded tests.

So, if you’re looking for exceptional read performance and don’t mind keeping a close eye on your workloads, S2D is a great option. But if you’re after consistent write and mixed I/O performance with better capacity efficiency, all in a rugged ‘fire-and-forge’ mode, StarWind VSAN is the way to go.

Stay tuned for our upcoming articles, where we’ll dive deeper into these solutions to give you a bigger picture of how they can fit into your IT strategy.