Search
StarWind is a hyperconverged (HCI) vendor with focus on Enterprise ROBO, SMB & Edge

ReFS: Performance

  • June 23, 2016
  • 18 min read
Anton Kolomyeytsev is StarWind CTO, Chief Architect & Co-Founder. Microsoft Most Valuable Professional [MVP] in Cluster 2014 & 2015. SMB3, NFS, iSCSI & iSER, NVMe over Fabrics.
Anton Kolomyeytsev is StarWind CTO, Chief Architect & Co-Founder. Microsoft Most Valuable Professional [MVP] in Cluster 2014 & 2015. SMB3, NFS, iSCSI & iSER, NVMe over Fabrics.

ReFS

ReFS (Resilient File System – https://msdn.microsoft.com/en-us/library/windows/desktop/hh848060%28v=vs.85%29.aspx) is a Microsoft file system, which ensures data integrity by means of resiliency to corruption (irrespective of software or hardware failures), increases data availability and scales to large data sets across various workloads. Its data protection feature is represented by the FileIntegrity option, which is responsible for file scanning and repair processes.

All ReFS metadata has a 64-bit checksum, which is stored independently of file data. Also, the file system can have its own checksum in a separate integrity stream. With FileIntegrity on, ReFS automatically checks data integrity by examining this checksum data. Its predecessors, in contrast, required administrators to manually run CHKDSK or similar utilities for this purpose. So, if FileIntegrity option is off, the system won’t “know” about the data corruption (if any), and if it’s on, then the system will be aware.

A testing has been run to see how ReFS performs under typical virtualization workload. It was divided into parts.

The first one was intended to study I/O behavior in ReFS with FileIntegrity option off and on. What has been observed with it is that with FileIntegrity option on or off, ReFS behaves differently: either as a conventional or log-structured file system. The full description of the test can be found here: https://slog.starwindsoftware.com/refs-virtualization-workloads-test-part-1/

This paper tells about the second part of the testing, which is intended to explore Resilient File System (ReFS) performance with FileIntegrity option off and on. It is the second part of ReFS performance testing.

Test

To study ReFS performance with FileIntegrity off/on we need to run series of experiments. We are planning to compare operation of the hard drive according to the following scenarios:

  • unformatted,
  • NTFS-formatted,
  • ReFS-formatted with FileIntegrity off
  • ReFS-formatted with FileIntegrity on.

For the workload we are going to use Iometer, which is an I/O subsystem measurement and characterization tool: http://www.iometer.org/ We’ll use Diskspd test later, as well (https://gallery.technet.microsoft.com/DiskSpd-a-robust-storage-6cd2f223).

Also, the workload will include 4 workers, 16 outstanding IOs, and 4K random write.

Hardware setup specs for testing:

Intel Xeon X3460 @ 2.8GHz

RAM 8GB

Boot: SATA WD5000AAKX – 500 GB

Test disk: (E:\): SATA WD1003FZEX – 1TB

Device manager

Testing RAW performance of Test disk hard drive with Iometer.

Iometer settings:

4 Workers

# of Outstanding I/Os – 16

4K random write

Testing time – 12 hours

This timespan will be sufficient for seeing how the drive reacts to the workload if it sustains continuous workload and the changes in its performance. Also, it will help to identify potential errors arising as a result of buildup.

Iometer

Iometer

Test results for Test disk:

Write IOPS – 154.444216

Write MBps – 0.632604

 

Diagrams.

For all workers and total value.

Iometer test results

For each worker separately:

Iometer test results

Iometer test results

Iometer test results

Iometer test results

Next, we format this SATA WD1003FZEX – 1TB into NTFS, test its performance with Iometer, and compare the resulting performance with one of the unformatted disks.

disk management

Iometer settings:

4 Workers

Maximum Disk Size – 1048576000 (500 GB)

# of Outstanding I/Os – 16

4K random write

Testing time – 12 hours.

Iometer

Iometer

Test results for Test disk (E drive formatted into NTFS file system):

Write IOPS – 181.859695

Write MBps – 0.744897

 

Diagrams.

For all workers and total value.

Iometer test results

For each worker separately.

Iometer test results

Iometer test results

Iometer test results

Iometer test results

Results for RAW and NTFS-formatted volumes

As we can see, the values and their changes for the RAW drive and this NTFS-formatted one are almost the same. Only the average performance value for the drive with NTFS is slightly higher, which can be explained either by some optimization being run by NTFS or with the system merging some adjacent writes, which is called write coalescing (https://en.wikipedia.org/wiki/Coalescing_(computer_science) ). Also, though patterns are random, something could still have been cached, and this can be another explanation.

So, NTFS-formatted volume and RAW volume performances are pretty much identical. Their requests sizes and requests sequences were the same. Being a conventional file system, NTFS doesn’t change requests sequence and requests pattern.

Afterwards, this SATA WD1003FZEX – 1TB drive has been formatted into ReFS.

Disk management

Checking FileIntegrity value for it:

Checking FileIntegrity PowerShell command

As we can see, it is off.

Performing a load testing of E drive (ReFS-formatted with FileIntegrity off) with Iometer.

Iometer settings:

4 Workers

Maximum Disk Size – 1048576000 (500 GB)

# of Outstanding I/Os – 16

4K random write

Testing time – 12 hours.

Iometer

Iometer

Checking FileIntegrity setting for test file Iometer (E:\iobw.tst)

Checking FileIntegrity setting for test file Iometer PowerShell command

As we can see, it is off.

ReFS with FileIntegrity off. Free disk space changes

During the test, we track the performance and changes in free disk space. Based on the observation results, we conclude that no performance failures occurred, and the results are nearly identic. No changes in free disk space occurred. This behavior is typical of conventional file systems: read-write is carried out on pre-allocated blocks, and the amount of free disk space doesn’t change. It’s one of the features of the conventional file systems – necessary, but not sufficient to describe ReFS as a totally conventional one in this mode.

Here’s what is displayed:

ReFS Properties

Checking the space taken by the test file:

Iometr test file

Test results

Write IOPS – 181.107504

Write MBps – 0.741816

 

Diagrams.

For all workers and total value.

Iometer test results

For each worker separately.

Iometer test results

Iometer test results

Iometer test results

Iometer test results

Results for ReFS-formatted volume with FileIntegrity off

The values found and their changes are identic to the values and changes for this drive RAW and this drive formatted into NTFS.

ReFS with FileIntegrity off behaves like NTFS: it doesn’t change the size of I/O blocks, but only passes the commands to the lower level. So, the pattern doesn’t change dramatically, as well. We may conclude here that ReFS with FileIntegrity off behaves much like a conventional file system.

Afterwards, we format E drive into ReFS again and turn FileIntegrity on for the data. Checking the status.

Checking status PowerShell command

As we can see, now FileIntegrity is on.

Performing a load testing of E drive (ReFS-formatted with FileIntegrity on) with Iometer.

Iometer settings:

4 Workers

Maximum Disk Size – 1048576000 (500 GB)

# of Outstanding I/Os – 16

4K random write

Testing time – 12 hours.

Iometer

Iometer

Checking FileIntegrity setting for Iometer (E:\iobw.tst) test file

Checking FileIntegrity setting PowerShell command

ReFS-formatted with FileIntegrity on. Free disk space changes

During the test, we track the performance and changes in free disk space. Based on the observation results, we can see that no changes in used space occurred, and free space fluctuates within 1 Gigabyte.

Free disk space changes, if they happen during file reading, mean, that the file system is log-structured. In the course of work, the system takes some additional space for internal housekeeping and gives it back afterward. So, with FileIntegrity on, ReFS behaves like a log-structured file system.

ReFS Properties

ReFS general Properties

Performance during free disk space changes

As for the performance results, at the moment of changes in the free space, the performance gradually decreases and finally drops to zero.

Iometer

Iometer

Iometer

Afterwards, the free space value changes, performance values regain and the process goes round.

Unexpectedly, during the testing (within about 15-30 minutes after the start) the Iometer froze.

Iometer

We decided to wait for 3 or 4 hours to give Iometer some time to get back to normal, but nothing changed, so we had to restart the system.

Test results.

The file with the general results is empty. Probably, that is because the test has never been completed.

Test results

Values are available in the file with time details. We’ll put them on the diagram.

 

Diagrams.

For all workers and total value.

Iometer test results

For each worker separately.

Iometer test results

Iometer test results

Iometer test results

Iometer test results

Results for ReFS-formatted volume with File Integrity on

As can be seen from the diagram, at the beginning of the testing (around 4 or 5 minutes), there’s a strong leap of performance, but, after the short-time increase, the values go back to nominal/standard ones, which were obtained at the earlier stages of testing, and remain the same for about 10 minutes, and afterwards the system freezes with zero results.

The similar test was carried out 3 or 4 times. In each case, the freezing repeated after a while. We failed to complete the performance test on ReFS with FileIntegrity on.

ReFS with FileIntegrity on changes sequence of commands, changes requests, I/O type, so it removed many random writes. As a result, there’s a strong leap of IOPS, then IOPS fall, the garbage collector is launched, and free disk space runs out. In the end of the test, free disk space drops to zero.

Conclusion

ReFS with FileIntegrity off performs like a conventional file system, identically to NTFS (https://en.wikipedia.org/wiki/NTFS), which preceded it. Typical of a regular file system, no changes in free disk space are observed because read-write takes place on pre-allocated blocks. I/O blocks size and the pattern don’t change, as well. This mode makes ReFS suitable for the modern high-capacity drives and huge files since no chkdsk or scrubber is active.

With FileIntegrity on, we observe fluctuations in free disk space, changes of a sequence of commands, changes of requests, of I/O size and pattern. This means that in this mode ReFS works like a Log-Structured File System (https://en.wikipedia.org/wiki/Log-structured_file_system). This sounds good for virtualization workloads because the system transforms multiple small random writes into bigger pages, which increases performance and prevents from the “I/O blender” effect. This issue is typical for virtualization and refers to an effect of dramatic performance degradation, which results from multiple virtualized workloads being merged into a stream of random I/O. Before LSFS appeared, solving this problem came expensive. Now we have LSFS (WAFL, CASL) and, as it turned out, ReFS can help, too.

So, the main conclusion to be done here is that ReFS with FileIntegrity on works like Log-Structured File System. Is this a good thing? The issue is that, when FileIntegrity’s on, it’s not quite clear which workload we’re going to deal with. There’s a number of other issues, as well, but we’d prefer to leave this topic for another day and another article. Stay tuned.

Hey! Found Anton’s insights useful? Looking for a cost-effective, high-performance, and easy-to-use hyperconverged platform?
Taras Shved
Taras Shved StarWind HCI Appliance Product Manager
Look no further! StarWind HCI Appliance (HCA) is a plug-and-play solution that combines compute, storage, networking, and virtualization software into a single easy-to-use hyperconverged platform. It's designed to significantly trim your IT costs and save valuable time. Interested in learning more? Book your StarWind HCA demo now to see it in action!