Reading Time: 2 minutes

Since version 4.x, VMware vSphere provides native API to perform image level backup of running VMs: VMware vSphere Storage APIs – Data Protection (formerly known as VMware vStorage APIs for Data Protection or VADP).

Those API are based on VMware snapshot to create a readable base VMDK disk that can be easily copied by a compatible backup solution using a full transfer or using incremental transfer with Changed Block Tracking (CBT) feature.

A backup product using VMware vSphere Storage APIs – Data Protection can backup vSphere virtual machines from a central backup server or virtual machine without requiring backup agents or requiring backup processing to be done inside each guest virtual machine on the ESX host. This offloads backup processing from ESX hosts and reduces costs by allowing each ESX host to run more virtual machines.

But those API are not always bug free… and can there can data corruption in specific build version and specific cases. Almost associated with the CBT based backup.

Here a brief table to summarize some build version affected by possible issues:

vShere versionSolutionIssues
8.0.2Update to ESXi 8.0 Update 2b, build 23305546 Possible CBT corruption after a virtual disk hot extend
KB 95965
6.7Follow KB instructionThe virtual disk is either corrupted or not a supported format when committing or creating snapshots
KB 2013520
6.5/6.7Upgrade to VMware VDDK 6.7 or later versionCBT reports larger area of changed blocks than expected if guest OS performed unmap on a disk
KB 59905
6.0.0Update and follow KB instructionsVMware vSphere 6 CBT issue
KB 2114076
KB 2136854
6.xApply laters updatesWhen users revert to a snapshot, it will automatically disable and enable CBT. If the snapshot is not a memory snapshot, enabling CBT will fail and CBT is disabled after reverting.
KB 71155
4.x/5.xApply patchesPossible backup data corruption after extending virtual machine VMDK file with Changed Block Tracking (CBT) enabled
KB 2090639

As you can notice there is actually on conditions that still can happen in an recent version of vSphere (without the lastest patch):

  • If you hot resize a VM disk on a vSphere 8.0.2 version, then CBT may have incorrect information. A fix has been released some days after the bug, and ESXi 8.0 Update 2b, build 23305546 solve that problem.
    Funny that a similar bug was introducted in old 4.x and 5.x versions.
    So safer solution is don’t perform hot resize, but resize VMDK only on powered off VMs.
    Follow KB 95965 to learn more!