How is NVMe-oF Doing? Part 4: “Who’s the fastest hand in the West”?

There was a race to bring NVMe-oF to Windows some time ago. Many tried, but I know that only StarWind and Chelsio succeeded so far. Now, it’s time to see who won this race! It is the fourth part of my NVMe-oF Initiators’ performance study. Before, I tested NVMe-oF initiators developed by Linux, Chelsio and StarWind. Here, the battle ends: which NVMe-oF initiator delivers the highest performance and which one Windows admins should use

INTRODUCTION

There was a race to bring NVMe-oF to Windows some time ago. Many tried, but I know that only StarWind and Chelsio succeeded so far. Now, it’s time to see who won this race!

It is the fourth part of my NVMe-oF Initiators’ performance study. Before, I tested NVMe-oF initiators developed by Linux, Chelsio and StarWind. Here, the battle ends: which NVMe-oF initiator delivers the highest performance and which one Windows admins should use?

RESULTS

RAM disk performance (all results)

I guess that it is good to have a table with RAM disk performance measured over loopback and locally. Just to see the maximum performance.

RAM disk performance (initiators only)

Here is the table to compare Initiators’ performance.

Intel Optane 900P performance (all results)

Now, let’s take a closer look at the efficacy of studied NVMe-oF Initiators for presenting flash over RDMA.

Intel Optane 900P performance (all results)

WHAT ABOUT THE LATENCY?

Performance is not only about IOPS. Here is the table demonstrating what the latency was like in all experiments.

DISCUSSION

I believe that we’ve seen enough to judge on NVMe-oF initiators’ performance. Linux NVMe-oF Initiator beats all other initiators under any workload. StarWind NVMe-oF Initiator took the second place. Chelsio NVMe-oF Initiator took the third place.

Let’s discuss how efficient can be presenting a RAM disk over RDMA in detail. In 64k blocks, StarWind NVMe-oF Initiator exhibited the same performance as Linux NVMe-oF Initiator. Interestingly, the former did slightly better in 64k sequential reading than Linux NVMe-oF Initiator. Chelsio NVMe-oF Initiator, in turn, showed only ¼ of Linux NVMe-oF Initiator performance in most of the tests.

Now, let’s see how the discussed NVMe-oF Initiators work with an NVMe drive (Intel Optane 900P). In 4k blocks, Linux NVMe-oF Initiator was doing better than any other Initiator. But, under some patterns, StarWind NVMe-oF Initiator was doing better than the solution developed by Linux. Under mixed loads in small blocks (4k sequential 50%read50%write), StarWind NVMe-oF Initiator performance was close to Linux NVMe-oF Initiator. For 4k sequential writing, StarWind NVMe-oF exhibited better performance than Linux NVMe-oF Initiator (385 254 and 257 360 IOPS respectively). Under that testing pattern, Chelsio NVMe-oF Initiator performance was close to what we saw for Linux NVMe-oF Initiator (254 299 and 257 360 IOPS respectively). All initiators exhibited similar performance in 64k blocks.

CONCLUSION

Linux NVMe-oF Initiator is the best solution to present an NVMe drive over the network so far. There was some serious work done to bring NVMe-oF to Windows; and StarWind NVMe-oF Initiator is the best solution for Windows so far. It just works great for any type of virtualized workload.