Reading Time: 3 minutes

In the last Veeam Community Forums Digest newsletter there is an interesting note about possible performance issues in VMware vSphere backup based on hot-add transport mode.

VMware has confirmed that the increased Hot Add times experienced by some Veeam customers who have installed Veeam Backup & Replication Update 3a and the reason is the latest Virtual Disk Development Kit (VDDK) 6.5 version used in U3a to support also vSphere 6.5 Update 2.

VDDK describes how to develop software using a virtual disk library that provides a set of system-call style interfaces for managing virtual disks on ESXi hosts.

Those interfaces are basically the VADP API, that describes how to write native backup and restore software that can be managed by vCenter Server for vSphere.

The VMware KB 55418 (Hotadd backup/restore takes time to open VMDKs in VDDK 6.5.2 and later versions) document this problem:

  • While using VDDK 6.5.2 and later versions, Hotadd backup/restore may take time to open VMDKs because of new functionality added to detect VMDK health status.
  • This issue is seen only with windows proxy machines as windows VDS (Virtual Disk Service) is used to check the VMDK health status before opening for backup/restore.

And actually, there is no fix currently in VDDK 6.5 branch. The good news is that the fix is already available in the newer VDDK 6.7 Express Patch 1 build.

Veeam has not yet decided what can be done for vSphere 6.5. One option could be just rolled back to using the previous version of VDDK 6.5 which does not have this bug yet but it’s not ideal from the security perspective. Another option could use the VDDK 6.7 EP1 also for processing vSphere 6.5 hosts (in addition to vSphere 6.7 hosts), but this can bring some possible bug and issues.

Is it only a Veeam problem? Not really… all VMware native backup programs that are using the latest VDDK 6.5.2 version are affected, and maybe you just don’t know (because the backup vendor does not necessarily declare which version is used).

Note: this issues has been fixed by VMware ONLY for 6.7 as reported in the VMware KB 55418 (Hotadd backup/restore takes time to open VMDKs in VDDK 6.5.2 and later versions). Veeam has address this issue in a different way, as explained in Veeam KB 2711.

Anyway, if you are using hot-add mode (for example for VSAN backup) check your backup job time and if they become too low consider if other transportort modes can be used. But remember that also network mode has some performance limitations.