Reading Time: 2 minutes

If you are using Veeam Backup & Replication (but also other backup solution) you can perform both backup and replication with the same tool.

But with latest version of vSphere you can probably notice a huge difference between backup and replication speed if you are using NBD transport mode.

And this issue is not related to Veeam, but it’s something builtin in the VDDK (Virtual Disk Development Kit) framework that is used by all the certified backup solution for VMware vSphere.

According to VMware KB 86269 (Impacted performance over NDB/NSBSSL)

Known issue where some customer’s are experiencing slow replication and restore over NDB

Customer’s may observe the backup job take more time when vSphere is upgraded to 7.0 and later.

https://kb.vmware.com/s/article/86269

VMware Engineering are aware of this issue, it is currently under investigation but at this point there isn’t a solution.

The cause, according to VMware KB 83401, could be related to:

  • There is high latency for storage IO.
  • The default AIO manager in NFC Server (which VDDK connects to in NBD mode) doesn’t have many threads to handle IOs asynchronously.

There are some mitigation but require to apply some configuration to VDDK framework, like adding those two options:

vixDiskLib.nfcAio.Session.BufSizeIn64KB=16
vixDiskLib.nfcAio.Session.BufCount=4

But in Veeam seems not possible add options to VDDK framework, the only documented option is just disable NFC AIO using a registry key.

And several other backup solution have the same problem that VDDK is just embedded and not configurable (to be honest Veritas is well documented how add those two options).

The best solutions are using other transport mode:

  • SAN mode (maybe using storage snapshot) on physical proxy… but remember that there are several limitation of using SAN mode with replica: only the first replica used SAN mode and only for thick disk!
  • Hot-Add mode using virtual proxy… there are some other limitations, but could be the most effective solution.

From some personal tests using latest Veeam B&R version, seems that Windows Server 2016 have more problem with NBD and replica… Windows Server 2019 has better performance, but still replication is slower than backup.

From the community posts and comments, seems that VDDK 7 has this issue in the Windows package and not in the Linux package.

Also the new VDDK 8 will resolve this issue (but probably will add several other bugs).

Share

Virtualization, Cloud and Storage Architect. Tech Field delegate. VMUG IT Co-Founder and board member. VMware VMTN Moderator and vExpert 2010-20 and vExpert Pro. Dell TechCenter Rockstar 2014-15. Microsoft MVP 2014-16. Veeam Vanguard 2015-19. Nutanix NTC 2014-20. Several certifications including: VCDX-DCV, VCP-DCV/DT/Cloud, VCAP-DCA/DCD/CIA/CID/DTA/DTD, MCSA, MCSE, MCITP, CCA, NPP.