Changed Block Tracking (CBT) is a VMware feature introduced in vSphere 4.0 in order to implement native incremental backup at source level. VMware VDAP uses this technology and so all backup and recovery software designed for VMware use it for speed the backup.
This mean that all the backup products that are using VDAP, will rely on VMware API and features, … and bugs.
In the past there were some bugs in CBT, especially in vSphere 6.x, for example this VMware vSphere 6 CBT issue.
Now, Veeam has discovered a new “bug” as described in the current weekly digest (a mailing list manage directly by Anton Gostev).
The “bug” happens if you revert a snapshot on a protected VM, after that the CBT API started to return invalid data.
Funny but it’s not a “bug”, but it’s a design issue: the CBT API does not support reverting snapshot on a VM as described in VMware KB 71155 (QueryChangedDiskAreas API returns incorrect value if you revert a snapshot).
This affect all backup product that are not using agent inside the guest OS. Note that only backup products that are using VDAP could be “VMware Certified”, but they don’t have to use all the VDAP features.
What are the possible solutions?
At least two:
- Don’t use VM snapshots in production environments at all: they may impact performance, they may overfill your datastores, and you can have data corruption in your backup.
- If you use VM snapshots, don’t revert VM snapshots. If you have to revert a snapshot, be sure to manually reset your CBT data. There is a nice script to do this (with several info) on Unitrends web site.