Resilient File System (ReFS) is a new Microsoft’s filesystem introduced for the first time in Windows Server 2012 with the intent of becoming the “next generation” filesystem after the “old” NTFS (first NTFS version was for Windows NT 3.1). Starting with Windows Server 2016, ReFS is the default filesystem for some workloads and some solutions (like S2D).
To provide greater resiliency for its metadata, the Resilient File System (ReFS) in Windows Server 2016 uses allocate-on-write semantics for all metadata updates. This means that ReFS never makes in-place updates to metadata. Instead, it makes all writes to newly allocated regions.
However, allocating-on-write causes ReFS to issue more metadata I/O to new regions of the volume than write-in-place file systems do. Additionally, ReFS uses block caching logic to cache its metadata in RAM. This is not as resource-efficient as file caching logic.
Together, the ReFS block caching logic and allocate-on-write semantics cause ReFS metadata streams to be large. ReFS uses the cache manager to create the metadata streams, and the cache manager lazily unmaps inactive views. In some situations, this lazy unmapping causes the active working set on the server to grow. This creates memory pressure that can cause poor performance.
This issue is more visible with 4K block size (but it affects volumes with bigger cluster as well, but having, for example, 64KB clusters makes memory and resource consumption less effective) and has been well documented on Veeam Forum (see this thread), considering that this solution has become a suggested option for Veeam targets on Windows Server 2016.
Finally the March 2017 cumulative update for Windows Server 2016 includes a fix of this issue, as documented in Microsoft KB 4016173. Just installing the update is not enough, because there are some registry changes that are needed, and also, after that, a server reboot.
But note that this fix seems to breaks Hyper-V backup for Veeam, so it’s better (for now) don’t install it on Hyper-V host, considering also that trying to remove it may cause the host to enter BSOD loop! It’s better to hold off installing it at least until Veeam (or Microsoft) will find a solution.