Reading Time: 3 minutes

Changed Block Tracking (CBT) is a VMware feature introduced in vSphere 4.0 in order to implement native incremental backup at source level. VMware VDAP uses this technology and so all backup and recovery software designed for VMware use it for speed the backup.

But last week a critical issue has been discovered on all vSphere 6  version (included the lates Update 1a) where Changed Block Tracking (CBT) data on ESXi 6 cannot be trusted.

This issue occurs due to an issue with CBT in the disklib area, this causes the change tracking information of I/Os that occur during snapshot consolidation to be lost. The main backup payload data is never lost and it is always written to the backend device. However, the corresponding change tracking information entries which occur during the consolidation task are missed. Subsequent calls to CBT APIs do not include these missed blocks, hence a backup based on this CBT data is inconsistent.

Currently, there is no resolution (patch ESXi600-201511401-BG released on Nov 25 should solve this issue). VMware have identified the root cause of this issue. This article will be updated when the ESXi patch resolving this issue is available.
To work around the issue, choose one of these options:

Veeam users have one more option, and that is to disable the use of CBT data in the advanced job settings. Your backups and replicas will remain incremental, but they will take longer, because the job will need to read the entire source disk to determine the changes. Disabling CBT is essential – otherwise, even Active Full backup may contain corruption, because CBT data is used there too to determine and skip zeroed regions of virtual disks. On the other hand, disabling CBT is sufficient to both prevent and remediate the issue, because this will make jobs physically compare latest state of disk in backup or replica with its actual state, and transfer any non-matching blocks over as a part of incremental backup (along with actually changed blocks), thus fixing any corruption that may already be in place.

So also after several months and one Update version, there still a reason to wait to move to vSphere 6.0 in production.

For more information on this issue see the KB2136854 (Backups with Changed Block Tracking can return incorrect changed sectors in ESXi 6.0).

Updated on November 25, 2015: patch ESXi600-201511401-BG should solve this issue. See also KB 2137546.

Share

Related Posts

  • How fix the VMware vSphere 6.0 CBT issue

    As you probably know VMware vSphere 6.0 had a critical issue con its Change Block Tracking (CBT) implementation that can impact all incremental backup with "VMware native" backup program (all agent-less implementation using the VMware VDAP API). This issue occurs due to an issue with…

  • Snapshot "issue" in VMware vSphere 6.0

    If you have installed (or upgraded to) VMware vSphere 6.0 and you still use the old legacy vSphere client you may have some false positives about VM snapshots. All VMs seems to have some snapshots, also if they don't (and using consolidate does not fix…

  • vSphere 5 download

    Finally the vSphere 5 download is now available on VMware site: http://downloads.vmware.com/d/info/datacenter_cloud_infrastructure/vmware_vsphere/5_0 VMware ESXi 5.0 (Build 469512) VMware vCenter 5.0 (Build 456005) Also the official documents is now available: http://www.vmware.com/support/pubs/vsphere-esxi-vcenter-server-pubs.html

Virtualization, Cloud and Storage Architect. Tech Field delegate. VMUG IT Co-Founder and board member. VMware VMTN Moderator and vExpert 2010-24. Dell TechCenter Rockstar 2014-15. Microsoft MVP 2014-16. Veeam Vanguard 2015-23. Nutanix NTC 2014-20. Several certifications including: VCDX-DCV, VCP-DCV/DT/Cloud, VCAP-DCA/DCD/CIA/CID/DTA/DTD, MCSA, MCSE, MCITP, CCA, NPP.