How is NVMe-oF doing? Part 3: StarWind NVMe-oF Initiator + Linux SPDK NVMe-oF Target

INTRODUCTION

Finally, I got the hands-on experience with StarWind NVMe-oF Initiator. I read that StarWind did a lot of work to bring NVMe-oF to Windows (it’s basically the first solution of its kind), so it’s quite interesting for me to see how their initiator works! In today’s post, I measure the performance of NVMe drive presented over Linux SPDK NVMe-oF Target while talking to it over StarWind NVMe-oF Initiator.

TOOLKIT USED

Linux SPDK RAM disk NVMe-oF Target ↔ StarWind NVMe-oF Initiator

Linux SPDK Optane NVMe-oF Target ↔ StarWind NVME-oF Initiator

Now, let’s talk more about the hardware configuration of my test environment. Here’s the hardware on the Target side (SPN77):

• Dell PowerEdge R730, CPU 2x Intel Xeon E5-2683 v3 CPU @ 2.00GHz, 128 GB
• Network: Mellanox Connect x4 100 Gbps
• Storage: Intel Optane 900P
• OS: CentOS 7.6 (Kernel 4.19.34) (Target)

Here’s what was inside the Initiator host (SPN76):

• Dell PowerEdge R730, CPU 2x Intel Xeon E5-2683 v3 @ 2.00GHz, 128 GB
• Network: Mellanox Connect x4 100 Gbps
• OS: Windows Server 2016

Today I measure how efficiently the storage of the Target host (SPN76) can be presented over RDMA by means of StarWind NVMe-oF Initiator+ Linux SPDK NVMe-oF Target. The latter was installed on the Target side, SPN77. Network throughput between hosts was measured with rPerf (RDMA) and iPerf (TCP).

MEASURING NETWORK BANDWIDTH

Before starting the actual tests, let’s see whether Mellanox ConnectX-4 can provide decent network throughput.

NOTE: CentOS starting with Kernel 4.19.34 comes with Mellanox drivers installed (i.e., there’s no need to install them manually). Here’s how to load Mellanox ConnectX-4 drivers.

Now, install Mellanox OFED for Windows (http://www.mellanox.com/page/products_dyn?product_family=32&mtag=windows_sw_drivers) on SPN76.

Next, I checked whether NIC-s in my setup support RDMA. I used the rPerf (https://www.starwindsoftware.com/resource-library/starwind-rperf-rdma-performance-benchmarking-tool) utility kit consisting of two utilities: rperf and iperf. In rPerf for Windows, they are called nd_rping and rd_rperf respectively. The former is a qualitative tool allowing to see whether hosts can talk over RDMA while the latter allows for the qualitative analysis of host connectivity.

Install rPerf on both servers and see whether hosts can talk over RDMA.

On the Initiator host, start the utility with the –s flag (i.e., server mode).

Run rping on Target (SPN77) in the client mode (-c flag) next.

Being set like that, SPN77 starts talking to SPN76 over RDMA. Although it looks that I have assigned the roles wrong (in my case, Target talks to Initiator), rping will still work fine as it just doesn’t care about Target and Initiator; NIC’s ability to talk over RDMA is the only thing that matters for this piece of software.

Here’s the output proving that there’s RDMA connection between the servers.

Now, let’s benchmark Mellanox ConnectX-4 throughput over TCP with iPerf. You can download this utility here: https://iperf.fr/iperf-download.php#windows.

Here’s the command for installing iPerf.

iPerf has to be installed on both hosts. One of them has to be run in client mode while another is started as a server. Here’s how to label one host as a client.

Find the command to run the utility in the server mode below.

Here’s the output showing what TCP throughput was like.

Next, RDMA connection was checked with rPerf. Below, find the output showing what network throughput was like when measured in 64k blocks.

(9845.82*8)/1024=76.9 Gbps

Let’s measure RDMA connection throughput in 4k blocks now.

(4265.90*8)/1024=33.32 Gbps

Discussion

The network won’t be a bottleneck. The observed RDMA and TCP network throughputs (77 Gbps and 94.6 Gbps respectively) were close enough to Mellanox ConnectX-4 network bandwidth, meaning that network doesn’t limit the underlying storage performance.

CONFIGURING THE TARGET AND INITIATOR

Install nvmecli

Install nvmecli on both servers using this command.

Start the Initiator on SPN77 and SPN76

Afterward, you can move to configuring a RAM disk.

Setting up a RAM disk

You need targetcli (http://linux-iscsi.org/wiki/Targetcli) to create a RAM disk. Find the command to install it below:

Next, run these commands to make sure that targetcli will be running even after a host reboot.

Create a 1 GB RAM disk with targetcli and present it as a block device.

Now, check whether the disk was created with Lsblk. Here’s the output after the disk has been successfully created.

RAM disk is listed as the /dev/sdb directory.

Setting up the Target

To start with, download SPDK (https://spdk.io/doc/about.html).

Here’s how a configuration retrieved from nvmf.conf looks like (find this file in spdk/etc/spdk/).

Now, take a look at the config file for Intel Optane 900P benchmarking.

Here’s the command to start the target:

Setting up the Initiator

Before you start the initiator, it is necessary to deploy the prepare_test_machine.cmd script. It installs certificates and sets the server into Test mode. Note that the host reboots shortly after running the script. You also need to disable integrity check to be able to install self-signed certificates.

Here’s the prepare_test_machine.cmd listing.

Now, you need to install the Initiator driver. You can do that only manually. See how that can be done below.

1. Go to Device Manager and press Add legacy hardware.

2. Next, tick the Install the hardware that I manually select from a list option.

3. Press Show All Devices afterward.

4. Open the Have Disk menu.

5. Specify the path to StarNVMeoF.inf.

6. If everything was done right, StarWind NVMe over Fabrics Storage Controller would be listed in the Model field.

7. Press Next.

8. Wait until the Add Hardware wizard finishes.

9. StarWind NVMe over Fabrics Storage Controller is now on the Storage controllers list.

Now, let’s start fine-tuning StarWind NVMe-oF Initiator.

1. Open StarNVMeoF_Ctrl.exe via CLI as administrator.

2. If you start StarNVMeoF_Ctrl.exe without any parameters, you’ll get the list of available commands.

3. Once target and initiator IP-s are specified, StarNVMeoF_Ctrl.exe discovery lists all available devices. Use the command below to enter the IP-s.

4. Next, specify target and initiator NQN (initiator NQN in my case is nqn.2008-08.com.starwindsoftware). Here’s the command allowing to do that:

Here are 2 more commands that may be interesting for you to know:

StarNVMeoF_Ctrl.exe list this command shows all the connected devices.

StarNVMeoF_Ctrl.exe remove disconnects the specific device from the initiator.

HOW I MEASURED EVERYTHING HERE

I think that it is a good idea to discuss how I carried out the measurements before moving to them.

Create a RAM disk with targetcli. Connect this disk as a local block device and benchmark its performance with FIO. RAM disk performance is going to be used as a reference only for the second step. Note that it is the maximum performance that can be observed for RAM disk in my setup.

Create an SPDK NVMe-oF target on the RAM disk (it is called Malloc in SPDK) on the Target host (SPN77). Connect the disk to Linux NVMe-oF Initiator located on the same host over loopback. Measure disk performance; that’s the new reference, i.e., the highest possible performance when the disk is presented to an initiator.

Create an SPDK NVMe-oF target on the RAM disk that resides on the Target side (SPN77). Present it over RDMA to the Initiator host (SPN76). Measure RAM disk performance over RDMA and compare it to the performance observed for a local RAM disk connected over loopback to the Initiator on the same host.

Connect Intel Optane 900P to SPN77 and benchmark it with FIO. This is the local drive performance that should be close to the value which one may find in vendor’s datasheet; no wonders that I use it as the ultimate reference.

On SPN77, present Intel Optane 900P to the local Linux NVMe-oF initiator by means of Linux SPDK NVMe-oF Target. That’s the reference that I use here to judge on StarWind NVMe-oF Initiator performance.

Measure NVMe drive performance while it is presented over the network. To do that, present Intel Optane 900P on SPN77 over RDMA to the Initiator on SPN76.

Herein, I used FIO (https://github.com/axboe/fio) for storage performance measurements.

Here are two ways of how you can install it. You can install it as a software package. Just use the command below.

Or, you can install it from the source using this set of commands:

BENCHMARKING THE RAM DISK

Picking the optimal test utility parameters

Before I start the real measurements, I’d like to find the optimal test utility parameters, i.e., such numjobs (number of treads) and iodepth (queue depth) values that ensure the best possible disk performance. To find these parameters, I measured 4k random reading performance. In my tests, I had the numjobs parameter fixed while varying iodepth. I run these measurements for various numbers of threads (1, 2, 4, 8). Below, find how the FIO listing looked like for varying queue depth under numjobs=1.

Here are the numbers I got.

Discussion

According to the plot above, numjobs = 8 iodepth = 4 are the optimal FIO parameters for testing RAM disk performance. Below, find the test utility listing.

RAM disk performance (connected via loopback)

RAM disk performance (presented over RDMA)

CAN I SQUEEZE ALL THE IOPS OUT OF AN INTEL OPTANE 900P?

Picking the optimal test utility parameters

Let’s find the best possible FIO settings. I ran a bunch of tests for different numbers of threads under varying queue depth (4k random read).

Here’s what performance is like.

Discussion

Under numjobs = 8 iodepth =4, I basically reached performance from Intel’s datasheet: https://ark.intel.com/content/www/us/en/ark/products/123628/intel-optane-ssd-900p-series-280gb-1-2-height-pcie-x4-20nm-3d-xpoint.html (see the screenshot below). This means that these test utility parameters are the optimal ones.

Intel Optane 900P performance (connected over loopback)

Intel Optane 900P performance (presented over RDMA)

RESULTS

RAM disk

Intel Optane results

Discussion

In 64k blocks, RAM disk, while being presented over RDMA, reached the same performance as when it was connected over loopback. In 4k blocks though, RAM disk, while being presented over RDMA, exhibited significantly lower performance than when it was connected over loopback (250K-300K IOPS less).

For Intel Optane 900P, things looked more or less the same. In 64k blocks, Intel Optane 900P provided the same performance over RDMA as when it was connected to the local target via loopback. In 4k blocks, the drive exhibited roughly 100 000 – 150 000 IOPS lower performance than while being connected locally.

WHAT ABOUT THE LATENCY?

Performance is just as important metric as the latency, so I think that this study cannot be considered complete without latency measurements. FIO settings: numjobs = 1 iodepth = 1.

RAM disk

Intel Optane

CONCLUSION

Today, I measured the performance of an NVMe drive presented over the network with Linux SPDK NVMe-oF Target + StarWind NVMe-oF Initiator for Windows. The main idea was to check whether a solution that brings NVMe-oF to Windows can unleash the whole potential of NVMe drives. StarWind NVMe-oF Initiator is a great solution allowing to enjoy the whole potential of NVMe drives.

In my next article, I sum up the results that I observed before and find out which NVMe-oF Initiator works better for presenting an NVMe SSD over RDMA.