DRBD/LINSTOR vs Ceph vs StarWind VSAN: Proxmox HCI Performance Comparison

Introduction

Hyperconverged infrastructure (HCI) has revolutionized the way organizations manage their IT environments. By blending compute, storage, and networking into a single platform, HCI offers simplicity, scalability, and cost-effectiveness.

But with so many HCI storage options available, choosing the right one can feel like navigating a maze. That’s where we come in. In this article, we’re going to take a deep dive into three popular HCI storage solutions, all running on a 2-node Proxmox cluster.

Here’s what we’ll be looking at:

• StarWind Virtual SAN (VSAN) NVMe-oF over RDMA/TCP: This solution promises fast performance with its NVMe technology, coupled with host mirroring and MDRAID-5.
• Linstor/DRBD over TCP: Using DRBD technology, this solution offers robust host mirroring and MDRAID-5, promising reliability and efficiency.
• Ceph: While not intended for production use in a 2-node setup, we’ll still explore Ceph’s capabilities for the sake of understanding its performance in this scenario.

Our goal? To give you the inside scoop on how these solutions perform in real-world situations. We’ll be running tests, crunching numbers, and providing practical insights to help you make informed decisions about your HCI storage needs.

So, whether you’re an IT pro, a system architect, or just curious about HCI storage, buckle up as we dive into the world of hyperconvergence and uncover the secrets behind these HCI storage solutions.

Solutions Overview

StarWind VSAN NVMe-oF over RDMA:

StarWind VSAN NVMe-oF over RDMA leverages Remote Direct Memory Access (RDMA) technology for high-speed data transfers between nodes. In this scenario, each Proxmox node is equipped with 5 NVMe drives, ensuring fast and efficient storage access. To enable RDMA compatibility for the Mellanox Network Interface Cards (NICs) on the Proxmox nodes, Single Root I/O Virtualization (SR-IOV) is configured. This allows the NICs to create Virtual Functions (VFs), which are then passed through to the StarWind Controller Virtual Machine (CVM). Within the StarWind CVM, the 5 NVMe drives from each Proxmox node are assembled into a single MDRAID5 array. This is achieved using the mdadm utility with specific creation parameters, including RAID level 5, a chunk size of 4K, an internal bitmap, and 5 RAID devices.

On top of the MDRAID5 array, two StarWind High Availability (HA) devices are created. This enhances fault tolerance and ensures continuous availability of data by replicating data between nodes.

StarWind VSAN NVMe-oF over TCP:

The configuration of this scenario is the same as StarWind VSAN NVMe-oF over RDMA. The only difference is in the network setup. Unlike the RDMA setup where SR-IOV is used for Mellanox NICs to enable RDMA compatibility, in the TCP configuration, SR-IOV is not configured. Instead, network communication is facilitated over TCP using Virtio network adapters. The StarWind CVM utilizes Virtio network adapters for both data and replication traffic. These adapters are specifically configured to support multiqueue, with 8 queues installed. This optimization ensures efficient network utilization and enhances overall performance.

Linstor/DRBD over TCP:

Similar to other configurations, the Linstor/DRBD over TCP scenario incorporates 5 NVMe drives on each Proxmox node, assembled into MDRAID5 arrays. On top of the MDRAID5 arrays, Linstor establishes a storage pool and resource group, configuring them with the “–place-count=2” parameter. This parameter determines the number of replicas for data redundancy and fault tolerance.

The Quorum Device Node was used as the diskless node with the “Quorum tiebreaker” feature for ensuring reliable cluster decision-making in case of network partitioning or node failure. Since the Linstor Diskless Node (Quorum Device Node) is connected to the Proxmox nodes via the 1GbE management network and doesn’t have RDMA compatibility, we couldn’t configure Linstor with RDMA transport.

Based on the performance recommendation, we have applied several tweaks, including adjustments to parameters such as max-buffers, sndbuf-size, and al-extents. These tweaks fine-tune the DRBD configuration for enhanced efficiency and throughput.

Ceph:

In this scenario, the focus is on evaluating the performance of a 2-node Ceph cluster, but don’t forget that such a setup is a non-production scenario. Instead, the aim is to gain insights into the potential performance capabilities of Ceph in this specific context, purely “for science purposes”.

Based on this blog, for setting up the Ceph cluster, each Proxmox node is configured to host two OSDs per NVMe drive, resulting in a total of 20 OSDs across the cluster (10 per node). This configuration is established using the command pveceph osd create /dev/nvme0n1 –osds-per-device=2.

Furthermore, a pool for RBD (RADOS Block Device) devices is created with a replicated rule, ensuring data redundancy and fault tolerance. The size and min_size parameters are set to 2 and 1 respectively, defining the replication factor. Additionally, the placement group count is set to 256 to optimize data distribution and balancing within the cluster.

For performance tuning, only one parameter, osd_mclock_profile, is adjusted. This parameter, set to high_client_ops, is intended to optimize Ceph’s handling of client operations, potentially enhancing overall system responsiveness and throughput.

Testbed overview:

The evaluation is performed on Supermicro SYS-220U-TNR nodes with Intel Xeon Platinum CPUs and Mellanox ConnectX-6 EN 200GbE NICs. Each node is equipped with 5x NVMe Micron 7450 MAX drives. Proxmox VE 8.1.3 serves as the hypervisor, with specific versions of StarWind VSAN, Linstor/DRBD, and Ceph.

Hardware:

Software:

StarWind CVM parameters:

Testing methodology

For our tests, we used a tool called FIO, which helps us measure performance in client/server mode. Here’s how we set things up: we had 16 virtual machines, each with 4 virtual CPUs, 8GB of RAM, and 3 raw virtual disks, each 100GB in size. Before running our tests, we made sure to fill these virtual disks with random data.

We tested various patterns to see how well our setup performs:

• 4k random read;
• 4k random read/write with a 70/30 ratio;
• 4k random write;
• 64k random read;
• 64k random write;
• 1M read;
• 1M write.

It’s important to note that before running some of our tests, we warmed up the virtual disks to ensure consistent results. For example, we ran the 4k random write pattern for 4 hours before testing the 4k random read/write 70/30 and 4k random write patterns. Similarly, we ran the 64k random write pattern for 2 hours before testing the 64k random write pattern.

We repeated each test 3 times to get an average result, and each test had a duration of 600 seconds for read tests and 1800 seconds for write tests. This way, we could be sure our results were reliable and representative.

Benchmarking local NVMe performance

Before starting our evaluation, we wanted to make sure the NVMe drive lived up to its vendor’s promises, so we ran a series of tests to see if its performance matched up. Here is the image with vendor-claimed performance:

Using the FIO utility in client/server mode, we checked how well the NVMe drive handled various workload patterns. The following results have been achieved:

The NVMe drive performed just as the vendor promised across all our tests. Whether it was handling small 4k reads or large 1M writes, it delivered on speed and consistency.

Tables with benchmark results

To provide a comprehensive analysis of the performance of different storage configurations, we conducted a series of benchmarks across various workload patterns. In addition to evaluating IOPS (Input/Output Operations per Second) and throughput, we also examined the performance dependency on CPU usage by introducing an additional parameter for 4k random read/write patterns: “IOPs per 1% CPU usage.”

This parameter was calculated using the formula:

IOPS per 1% CPU usage = IOPS / Proxmox Node count / Node CPU usage

Where:

• IOPS represents the number of I/O operations per second for each pattern.
• Proxmox Node count is 2 in our case (indicating the number of Proxmox nodes).
• Node CPU usage denotes the CPU usage of one Proxmox node during the test.

By incorporating this additional metric, we aimed to provide deeper insights into how CPU usage correlates with IOPS, offering a more nuanced understanding of performance characteristics.

Now let’s delve into the detailed benchmark results for each storage configuration.

StarWind VSAN NVMe-oF over RDMA

For the StarWind VSAN NVMe-oF over RDMA configuration, the performance was evaluated across multiple workload patterns. Notably, the 4k random read pattern showcased significant scalability with increasing IOdepth, reaching up to 4.8 million IOPS with an IOdepth of 64. Similarly, the 4k random read/write (70%/30%) pattern exhibited considerable performance, achieving up to 1.2 million IOPS with an IOdepth of 64. These results indicate efficient utilization of resources and high throughput capabilities in demanding scenarios.

StarWind VSAN NVMe-oF over TCP

In the StarWind VSAN NVMe-oF over TCP setup, the performance remained competitive, but with slightly lower IOPS compared to the RDMA configuration. However, the system demonstrated robust scalability, with the 4k random read pattern achieving over 2.3 million IOPS at an IOdepth of 32. The 4k random read/write (70%/30%) pattern also exhibited impressive performance, surpassing 887,000 IOPS at an IOdepth of 64. These results indicate the adaptability of the TCP protocol for NVMe-oF communication, providing high throughput and low latency.

Linstor/DRBD over TCP

In the Linstor/DRBD over TCP configuration, the system showcased commendable performance across various workload patterns. The 4k random read pattern demonstrated consistent scalability, achieving up to 1.5 million IOPS with an IOdepth of 32. Additionally, the 4k random read/write (70%/30%) pattern exhibited substantial performance, surpassing 1 million IOPS with an IOdepth of 64. These results highlight the effectiveness of the DRBD solution over TCP for storage replication, delivering high throughput and low latency.

Ceph

In the Ceph configuration, the performance was evaluated across different workload patterns, demonstrating the system’s capability to handle diverse workloads. Benchmark results show solid performance, though slightly lower compared to other solutions in some scenarios. For example, in the 4k random read pattern, IOPS ranged from 381,000 to 406,000, depending on the number of jobs and IOdepth.

Visualizing benchmarking results in charts

With all benchmarks completed and data collected, let’s use graphs to easily compare the results.

Figure 1 shows the number of Input/Output Operations Per Second (IOPS) achieved during 4k random read operations with Numjobs = 3. You can see that the performance of StarWind VSAN NVMe-oF over RDMA and StarWind VSAN NVMe-oF over TCP increases steadily as the queue depth increases, up to a point. This suggests that StarWind VSAN can benefit from queuing additional I/O requests to achieve better performance. Meanwhile, both Linstor/DRBD over TCP and Ceph reach their peak performance straight away. Increasing the queue depth for them only results in higher latency without performance growth.

StarWind VSAN NVMe-oF over RDMA raises the bar, showing a significant performance increase starting from queue depth = 8. At an IOdepth of 128, StarWind VSAN NVMe-oF over RDMA achieves 4,664,000 IOPS, while StarWind VSAN NVMe-oF over TCP achieves 2,944,000 IOPS. The performance gap widens even further when compared with Linstor/DRBD over TCP, where StarWind VSAN NVMe-oF over RDMA outperforms it by around 214.1%, compared to Linstor’s 1,483,000 IOPS. Against Ceph, the difference is even bigger, with StarWind VSAN NVMe-oF over RDMA surpassing Ceph’s 406,000 IOPS by more than x10 times.

Figure 2 compares the latency when reading 4k data blocks. Not surprisingly, StarWind VSAN NVMe-oF over RDMA exhibits the lowest latency among the tested solutions, with an average result of 0.5 milliseconds (ms) and a minimum of 0.168 ms. Moreover, for both RDMA and TCP configurations, StarWind VSAN NVMe-oF maintains lower latency compared to other solutions across all queue depths. In contrast, Linstor/DRBD over TCP and Ceph demonstrate a higher average latency increase, with Ceph being the slowest of all contenders.

CPU utilization efficiency is also a very important factor, as it is directly related to the cost-efficiency of the resulting solution, especially in a hyperconverged infrastructure (HCI).

Figure 3 shows IOPS per 1% CPU usage during a 4k random read test. StarWind VSAN NVMe-oF over RDMA and TCP demonstrate the most efficient CPU utilization, followed by Linstor/DRBD over TCP. Ceph lags behind the other solutions in terms of CPU efficiency, exhibiting the lowest IOPS per 1% CPU usage values, ranging from 2,886 to 3,076, and suggesting room for improvement in CPU resource utilization. At the same time, StarWind VSAN NVMe-oF over RDMA demonstrates notable performance improvements over other configurations when comparing their highest values: outperforming its TCP counterpart by approximately 52.67%, Linstor/DRBD over TCP by about 118.92%, and Ceph by impressive x13 times.

Pure read or write workloads don’t occur all the time in real production scenarios, which is why benchmarking storage performance with a mixed 70/30 read-write pattern is important to get the full picture.

Figure 4 demonstrates the number of IOPS achieved during 70%/30% 4k random read/write operations with Numjobs = 3. Here, StarWind VSAN NVMe-oF over RDMA again takes the lead, achieving the maximum result of 1,199,000 IOPS and outperforming other storage solutions, especially in tests with lower IOdepths.

Interestingly, Linstor/DRBD handled mixed workloads better than other solutions, achieving remarkable results. Specifically, when compared head-to-head with StarWind VSAN NVMe-oF over TCP, it scored 1,088,000 IOPS, gaining a solid advantage of 22.6% over the maximum 887,000 IOPS achieved by StarWind over TCP.

Ceph, on the other hand, consistently showed lower performance across the board, achieving a maximum of 270,000 IOPS.

Delving into latency dynamics, Figure 5 scrutinizes the delay times during 4k random read/write operations at a 70%/30% ratio with Numjobs = 3. Again, StarWind VSAN NVMe-oF over RDMA stands out with notably lower latency throughout the test, with Linstor/DRBD taking second place.

Figure 6 explores the efficiency of IOPS relative to CPU utilization during the same mixed workload. Here, StarWind VSAN NVMe-oF over RDMA demonstrates superior efficiency during the ‘heaviest’ IOdepth 64 pattern, delivering better IOPS per 1% CPU usage by 35.17% over its TCP version, 103.0% over Linstor/DRBD over TCP, and a striking 834.5% over Ceph, marking a significant performance advantage.

Interestingly, when comparing the ‘lighter’ IOdepth patterns of 8 and 16, Linstor/DRBD takes the lead over StarWind’s NVMe-oF over TCP, with an average 25% advantage in CPU utilization. However, this advantage is completely mitigated during the ‘heavier’ tests within the IOdepth 32 pattern. Moreover, StarWind VSAN NVMe-oF over TCP regains the lead, achieving 50.17% better CPU utilization than Linstor/DRBD during IOdepth 64 tests. Fundamentally, this means that StarWind VSAN NVMe-oF over TCP maintains the same level of CPU usage efficiency across all patterns, while Linstor/DRBD starts wasting more CPU resources on heavier workloads.

Figure 7 presents the IOPS achieved during 4k random write operations, revealing that StarWind VSAN NVMe-oF over RDMA not only tops its TCP configuration by about 29% but also surpasses Linstor/DRBD by 22% and Ceph by 115%, showcasing the benefits of RDMA in write-intensive scenarios.

Once again, the close competition between StarWind VSAN NVMe-oF over TCP and Linstor/DRBD continues here, with StarWind VSAN showing a 27.6% advantage over DRBD at IOdepth=4, going neck-to-neck during IOdepth=8, and finally, falling behind at heavier workloads, scoring 5.9% lower than DRBD. What a race!

In assessing write latencies, Figure 8 confirms the previous IOPS results, with StarWind VSAN NVMe-oF over RDMA leading in minimizing delays and outperforming other solutions across the board.

The close race between Linstor/DRBD and StarWind VSAN’s TCP configuration is also prominent here.

Efficiency takes center stage in Figure 9, which measures IOPS per 1% CPU usage during a 4k random write test. StarWind VSAN NVMe-oF over RDMA demonstrates remarkable CPU utilization, significantly outstripping other configurations and illustrating its ability to deliver higher performance with less resource consumption. It is more efficient by approximately 34.95% over StarWind VSAN NVMe-oF over TCP, 22.75% over Linstor/DRBD over TCP, and a dramatic 536.07% over Ceph in 32 queue depth tests.

What’s even more interesting, Linstor/DRBD takes a respectable second place, outperforming StarWind VSAN NVMe-oF over TCP configuration by 9.96% in IOdepth=32 tests.

Figure 10 illustrates the throughput for random reads in 64KB blocks. During this test, Linstor/DRBD over TCP emerges as a standout performer, achieving about 22.17% higher throughput than StarWind VSAN NVMe-oF over RDMA, 63.23% over StarWind VSAN NVMe-oF over TCP, and a whopping 182.27% over Ceph.

Linstor/DRBD’s high performance is attributed to its read-balancing feature, available since version 8.4.1, which balances read requests between the primary and secondary nodes. This feature allows you to define the read policy. In our testing, we used the default read-balancing policy, prefer-local, as it provided maximum performance through local reading and avoided the TCP stack usage. In your case, you can use another policy, such as read striping, round-robin, least-pending, and when-congested-remote, and your performance might improve with different parameters. If you want to explore this topic in more detail, please read here.

Figure 11 compares latency during 64K random reads, where Linstor/DRBD over TCP (surprise surprise) consistently shows the lowest latency, emphasizing its efficiency in reading large-sized data blocks. In IOdepth=16 tests, Linstor/DRBD over TCP demonstrates 18.76% lower latency compared to StarWind VSAN NVMe-oF over RDMA, about 38.17% lower than StarWind VSAN NVMe-oF over TCP, and approximately 64.57% lower than Ceph.

CPU usage during 64K random reads is analyzed in Figure 12, showcasing Linstor/DRBD over TCP with the most efficient CPU utilization. Compared to every other solution, Linstor/DRBD over TCP maintains CPU usage reductions by an average of 60.41% during IOdepth=2 tests, highlighting its efficiency on lighter workloads. Interestingly, the difference in efficiency decreases with heavier workloads, with Linstor/DRBD being more efficient than StarWind VSAN NVMe-oF over RDMA by 23.07%, 35.48% more efficient than StarWind VSAN NVMe-oF over TCP, and 57.89% more efficient than Ceph.

The throughput for 64K random writes is presented in Figure 13. While Linstor/DRBD over TCP and Ceph demonstrate higher throughput values at higher IOdepths, StarWind VSAN NVMe-oF over RDMA provides better throughput at lower IOdepth tests. At an IOdepth of 4, Linstor/DRBD surpasses StarWind VSAN NVMe-oF over RDMA by 13.3%, and its TCP configuration by 17.7%. Meanwhile, at IOdepth = 1, StarWind VSAN NVMe-oF over RDMA shows 29.04% better throughput compared to DRBD and 88.94% better than Ceph. Meanwhile, StarWind VSAN in TCP configuration shows 2.50% lower throughput than DRBD over TCP in IOdepth=1 tests and a 15.03% decrease in IOdepth=4 tests.

It’s also interesting how Ceph’s throughput ramps up as the IOdepth increases. This suggests that Ceph can handle more large-sized requests in parallel, reducing individual queuing times and leading to higher overall throughput.

In Figure 14, the latency of 64K random writes presents the same pattern as seen in previous throughput figures. StarWind VSAN NVMe-oF over RDMA offers the fastest response times at lower IOdepths. However, at an IOdepth of 4, Linstor/DRBD over TCP takes the lead, showing a lower latency of 2.49 ms. Ceph also does a great job at higher IOdepths, showing similar results to DRBD.

Let’s move on to Figure 15, which details CPU usage during 64K random writes. Linstor/DRBD over TCP comes out on top in this race, demonstrating the most efficient use of CPU resources across all tested configurations. Even at higher IOdepths, it showcases up to 16.66% lower CPU utilization compared to StarWind VSAN NVMe-oF over RDMA, 23.07% lower than VSAN over TCP, and 57.44% lower than Ceph.

The throughput results for 1024k reads, depicted in Figure 16, show that StarWind VSAN NVMe-oF over RDMA significantly outperforms all competitors from IOdepth 1 to 4. However, at IOdepth 8, Linstor/DRBD takes a slight edge, outperforming StarWind VSAN NVMe-oF over RDMA by 4.08%. StarWind VSAN’s TCP configuration shows 29.6% better performance than Linstor/DRBD at IOdepth=1, but at higher IOdepths, DRBD regains the edge and surpasses StarWind VSAN NVMe-oF over TCP by 44.54% at IOdepth 8. Once again, DRBD’s performance lead is attributed to its read-balancing feature and the ‘prefer-local policy’ chosen for our testing. Ceph, on the other hand, starts much slower than the others, but shows remarkable results at IOdepth 8, surpassing StarWind VSAN’s TCP configuration and falling behind DRBD and StarWind VSAN NVMe-oF by 13.74% and 9.27% respectively.

Latency during the 1024k read test is explored in Figure 17, which reflects the same pattern observed in previous throughput figures. StarWind VSAN NVMe-oF over RDMA offers the fastest response times at IOdepths 1 to 4. However, at IOdepth 8, Linstor/DRBD over TCP takes the lead, demonstrating lower latency across the board.

CPU usage during 1024k reads is recorded in Figure 18, where both StarWind configurations exhibit higher resource use compared to competitors. Remarkably, Linstor/DRBD over TCP uses considerably less CPU, even at the maximum IOdepth—up to 50% less compared to StarWind VSAN NVMe-oF over RDMA. Ceph’s average CPU utilization is 20%, placing it around the middle of the pack when compared to the other three configurations, taking into account the respective throughput figures.

Figure 19 examines 1024k sequential write throughput, highlighting Ceph’s superior performance, especially as IOdepth increases. It surpasses Linstor/DRBD over TCP by approximately 148.95% and StarWind VSAN NVMe-oF over RDMA by about 196.67%, showcasing its effectiveness in handling large write operations. Similarly, against StarWind VSAN NVMe-oF over TCP, Ceph exhibits a substantial lead of roughly 208.64%.

It’s important to note that this level of Ceph’s performance is largely due to its operation in a mirror (RAID 1) configuration over the network in our case, while StarWind and Linstor/DRBD operate in RAID 51 (local RAID 5 combined with RAID 1 over the network).

Figure 20 presents the same pattern, with Ceph leading at all IOdepths and showing lower latency than other competitors.

Lastly, Figure 21 focuses on CPU usage during 1024k writes. Linstor/DRBD over TCP stands out for its minimal CPU demand, demonstrating an improvement of about 34.04% in CPU efficiency over StarWind VSAN NVMe-oF over RDMA. Interestingly, both the RDMA and TCP configurations of StarWind VSAN show similar CPU usage figures.

Conclusion

Our in-depth benchmarking analysis has shed light on how different storage setups perform across various tasks. We’ve seen that StarWind VSAN NVMe-oF over RDMA delivers impressive performance in 4k workloads, significantly outperforming competitors in terms of both performance and lower CPU utilization.

Meanwhile, StarWind VSAN NVMe-oF over TCP also shows commendable performance in 4k workloads, although it slightly trails its RDMA counterpart and begins to lag behind Linstor in some mixed and large-sized, write-intensive tests.

Linstor/DRBD over TCP holds its own as well, particularly in mixed and large-size workloads such as 64K and 1024K. In some write tests, Linstor even outperformed StarWind VSAN over RDMA, which is a notable achievement for a TCP-based solution.

On the other hand, Ceph shows mixed results. While it excels at writing 1024K blocks, it falls short in almost all other scenarios, particularly in read workloads, and it lags behind in CPU efficiency.

The simple conclusion is that StarWind VSAN is perfect for running most kinds of virtualized workloads: general-use VMs, VDI, databases, and transaction-intensive tasks. Linstor/DRBD is also a good choice for most virtualization workloads, especially in environments where larger sequential reads and writes are typical. Ceph, meanwhile, is a good option for running data warehousing and analytics applications where 4K-64K IOPS are less relevant than high throughput during large sequential reads and writes. It’s also well-suited for environments dealing with video streaming and large file processing.

Ultimately, storage performance is not the sole factor to consider when building an HCI infrastructure. Ease of configuration and maintenance, necessary data protection and optimization features, commercial support, and other factors are also very important. Stay tuned, and we will help you get a bigger picture in our upcoming articles.