ReFS (Resilient File System – https://msdn.microsoft.com/en-us/library/windows/desktop/hh848060%28v=vs.85%29.aspx) is a Microsoft file system, which ensures data integrity by means of resiliency to corruption (irrespective of software or hardware failures), increases data availability and scales to large data sets across various workloads. Its data protection feature is represented by the FileIntegrity option, which is responsible for file scanning and repair processes.
All ReFS metadata has a 64-bit checksum, which is stored independently of file data. Also, the file system can have its own checksum in a separate integrity stream. With FileIntegrity on, ReFS automatically checks data integrity by examining this checksum data. Its predecessors, in contrast, required administrators to manually run CHKDSK or similar utilities for this purpose. So, if FileIntegrity option is off, the system won’t “know” about the data corruption (if any), and if it’s on, then the system will be aware.
A testing has been run to see how ReFS performs under typical virtualization workload. It was divided into parts.
The first one was intended to study I/O behavior in ReFS with FileIntegrity option off and on. What has been observed with it is that with FileIntegrity option on or off, ReFS behaves differently: either as a conventional or log-structured file system. The full description of the test can be found here: https://slog.starwindsoftware.com/refs-virtualization-workloads-test-part-1/
This paper tells about the second part of the testing, which is intended to explore Resilient File System (ReFS) performance with FileIntegrity option off and on. It is the second part of ReFS performance testing.
Test
To study ReFS performance with FileIntegrity off/on we need to run series of experiments. We are planning to compare operation of the hard drive according to the following scenarios:
- unformatted,
- NTFS-formatted,
- ReFS-formatted with FileIntegrity off
- ReFS-formatted with FileIntegrity on.
For the workload we are going to use Iometer, which is an I/O subsystem measurement and characterization tool: http://www.iometer.org/ We’ll use Diskspd test later, as well (https://gallery.technet.microsoft.com/DiskSpd-a-robust-storage-6cd2f223).
Also, the workload will include 4 workers, 16 outstanding IOs, and 4K random write.
Hardware setup specs for testing:
Intel Xeon X3460 @ 2.8GHz
RAM 8GB
Boot: SATA WD5000AAKX – 500 GB
Test disk: (E:\): SATA WD1003FZEX – 1TB
Testing RAW performance of Test disk hard drive with Iometer.
Iometer settings:
4 Workers
# of Outstanding I/Os – 16
4K random write
Testing time – 12 hours
This timespan will be sufficient for seeing how the drive reacts to the workload if it sustains continuous workload and the changes in its performance. Also, it will help to identify potential errors arising as a result of buildup.
Test results for Test disk:
Write IOPS – 154.444216
Write MBps – 0.632604
Diagrams.
For all workers and total value.
For each worker separately:
Next, we format this SATA WD1003FZEX – 1TB into NTFS, test its performance with Iometer, and compare the resulting performance with one of the unformatted disks.
Iometer settings:
4 Workers
Maximum Disk Size – 1048576000 (500 GB)
# of Outstanding I/Os – 16
4K random write
Testing time – 12 hours.
Test results for Test disk (E drive formatted into NTFS file system):
Write IOPS – 181.859695
Write MBps – 0.744897
Diagrams.
For all workers and total value.
For each worker separately.
Results for RAW and NTFS-formatted volumes
As we can see, the values and their changes for the RAW drive and this NTFS-formatted one are almost the same. Only the average performance value for the drive with NTFS is slightly higher, which can be explained either by some optimization being run by NTFS or with the system merging some adjacent writes, which is called write coalescing (https://en.wikipedia.org/wiki/Coalescing_(computer_science) ). Also, though patterns are random, something could still have been cached, and this can be another explanation.
So, NTFS-formatted volume and RAW volume performances are pretty much identical. Their requests sizes and requests sequences were the same. Being a conventional file system, NTFS doesn’t change requests sequence and requests pattern.
Afterwards, this SATA WD1003FZEX – 1TB drive has been formatted into ReFS.
Checking FileIntegrity value for it:
As we can see, it is off.
Performing a load testing of E drive (ReFS-formatted with FileIntegrity off) with Iometer.
Iometer settings:
4 Workers
Maximum Disk Size – 1048576000 (500 GB)
# of Outstanding I/Os – 16
4K random write
Testing time – 12 hours.
Checking FileIntegrity setting for test file Iometer (E:\iobw.tst)
As we can see, it is off.
ReFS with FileIntegrity off. Free disk space changes
During the test, we track the performance and changes in free disk space. Based on the observation results, we conclude that no performance failures occurred, and the results are nearly identic. No changes in free disk space occurred. This behavior is typical of conventional file systems: read-write is carried out on pre-allocated blocks, and the amount of free disk space doesn’t change. It’s one of the features of the conventional file systems – necessary, but not sufficient to describe ReFS as a totally conventional one in this mode.
Here’s what is displayed:
Checking the space taken by the test file:
Test results
Write IOPS – 181.107504
Write MBps – 0.741816
Diagrams.
For all workers and total value.
For each worker separately.
Results for ReFS-formatted volume with FileIntegrity off
The values found and their changes are identic to the values and changes for this drive RAW and this drive formatted into NTFS.
ReFS with FileIntegrity off behaves like NTFS: it doesn’t change the size of I/O blocks, but only passes the commands to the lower level. So, the pattern doesn’t change dramatically, as well. We may conclude here that ReFS with FileIntegrity off behaves much like a conventional file system.
Afterwards, we format E drive into ReFS again and turn FileIntegrity on for the data. Checking the status.
As we can see, now FileIntegrity is on.
Performing a load testing of E drive (ReFS-formatted with FileIntegrity on) with Iometer.
Iometer settings:
4 Workers
Maximum Disk Size – 1048576000 (500 GB)
# of Outstanding I/Os – 16
4K random write
Testing time – 12 hours.
Checking FileIntegrity setting for Iometer (E:\iobw.tst) test file
ReFS-formatted with FileIntegrity on. Free disk space changes
During the test, we track the performance and changes in free disk space. Based on the observation results, we can see that no changes in used space occurred, and free space fluctuates within 1 Gigabyte.
Free disk space changes, if they happen during file reading, mean, that the file system is log-structured. In the course of work, the system takes some additional space for internal housekeeping and gives it back afterward. So, with FileIntegrity on, ReFS behaves like a log-structured file system.
Performance during free disk space changes
As for the performance results, at the moment of changes in the free space, the performance gradually decreases and finally drops to zero.
Afterwards, the free space value changes, performance values regain and the process goes round.
Unexpectedly, during the testing (within about 15-30 minutes after the start) the Iometer froze.
We decided to wait for 3 or 4 hours to give Iometer some time to get back to normal, but nothing changed, so we had to restart the system.
Test results.
The file with the general results is empty. Probably, that is because the test has never been completed.
Values are available in the file with time details. We’ll put them on the diagram.
Diagrams.
For all workers and total value.
For each worker separately.
Results for ReFS-formatted volume with File Integrity on
As can be seen from the diagram, at the beginning of the testing (around 4 or 5 minutes), there’s a strong leap of performance, but, after the short-time increase, the values go back to nominal/standard ones, which were obtained at the earlier stages of testing, and remain the same for about 10 minutes, and afterwards the system freezes with zero results.
The similar test was carried out 3 or 4 times. In each case, the freezing repeated after a while. We failed to complete the performance test on ReFS with FileIntegrity on.
ReFS with FileIntegrity on changes sequence of commands, changes requests, I/O type, so it removed many random writes. As a result, there’s a strong leap of IOPS, then IOPS fall, the garbage collector is launched, and free disk space runs out. In the end of the test, free disk space drops to zero.
Conclusion
ReFS with FileIntegrity off performs like a conventional file system, identically to NTFS (https://en.wikipedia.org/wiki/NTFS), which preceded it. Typical of a regular file system, no changes in free disk space are observed because read-write takes place on pre-allocated blocks. I/O blocks size and the pattern don’t change, as well. This mode makes ReFS suitable for the modern high-capacity drives and huge files since no chkdsk or scrubber is active.
With FileIntegrity on, we observe fluctuations in free disk space, changes of a sequence of commands, changes of requests, of I/O size and pattern. This means that in this mode ReFS works like a Log-Structured File System (https://en.wikipedia.org/wiki/Log-structured_file_system). This sounds good for virtualization workloads because the system transforms multiple small random writes into bigger pages, which increases performance and prevents from the “I/O blender” effect. This issue is typical for virtualization and refers to an effect of dramatic performance degradation, which results from multiple virtualized workloads being merged into a stream of random I/O. Before LSFS appeared, solving this problem came expensive. Now we have LSFS (WAFL, CASL) and, as it turned out, ReFS can help, too.
So, the main conclusion to be done here is that ReFS with FileIntegrity on works like Log-Structured File System. Is this a good thing? The issue is that, when FileIntegrity’s on, it’s not quite clear which workload we’re going to deal with. There’s a number of other issues, as well, but we’d prefer to leave this topic for another day and another article. Stay tuned.
