If you have tried to apply the VMware vSphere 6.7 Update 3 patch on a DellEMC PowerEdge server, you may have some kind of performance issues.
The vCenter become very slow and some basic operations, including vMotion are very very slow (also more than 100 times!).
VMs performance are not affected, but if you have vSAN this can be a huge problem not only in managebility but also in system stability, because vCenter will notice several issues (including time screw between vCenter and hosts).
There was a thread on Spiceworks about this behavior, but finally has been confirmed also by DellEMC with this page.
Note that also if it seems a NIC driver issue, it isn’t! The reason is different.
The following symptoms may occur once a Dell server is upgraded to VMware ESXi 6.7 Update 3 if Dell OpenManage Server Administrator (OMSA) 9.3.0 is also installed:
- Tasks failing/timing-out on the host such as vMotion, snapshots, maintenance-mode, vm-power-on
- Host disconnects from vCenter as soon as you start one of the above tasks
- ESXi Embedded Host Client is not accessible
- Esxcli commands are unresponsive.
The following event may be noted in the main ESXi syslog:sfcb-dcim[2235472]: doLoadProvider: dlopen(/usr/lib/cim/libcmpiTunnelProvider.so) failed, dlerror: libcmpiCppImpl.so.0: cannot open shared object file: No such file or directory
The reason is because OMSA 9.3.0 released before ESXi 6.7 Update 2 or Update 3 were released and is therefore unsupported! In fact it was designed for Update 1 only.
Dell EMC Engineering will address this issue with full support in OMSA 9.3.1 releasing soon and this article will be updated with download instructions.
At this time there are two solutions:
- Roll back to vSphere 6.7 Update 2 (where OMSA 9.3.0 works perfectly, also if not supported). If you have not installed other pathes, this operation could be really easy: just reboot and choose Shift + R to load the alternative bank with the previous installation
- Remove the OpenManage package, unless a new version will be available.
To remove OpenManage use this command (from the ESXi command line):
esxcli software vib remove --vibname=OpenManage
You need also to reboot your ESXi to fix the issue.
Note that this can cause another issue related to hardware monitoring.
The host events log could be filled fast with similar lines:
In this case the VMware KB 74607 (Excessive Hardware health alarms being triggered for “Sensor -1 type” on ESXi hosts running vSphere 6.7 U3) will provide another temporary solution.
This is a known issue identified by VMware Engineering affecting vSphere ESXi 6.7 U3 build-14320388.
The functional impact should be minimal as the alarms getting triggered are not signifying any issue with the hardware. But these periodic alarms can result in excessive emails alerts if the user subscribed to email notification for Host hardware sensor state alarm in vCenter. Additionally, these events can result in the vCenter database increasing in size leading to insufficient disk space issues such as vCenter unavailability once the SEAT partition size goes above the 95% threshold.
And anyway, this issue can slow down your vCenter.
One possible workaround to avoid this issue is to disable the WBEM services.
To disable WBEM services on an ESX host use the following command:
esxcli system wbem set --enable false
To re-enable the CIM agent on the ESXi host, run this command:
esxcli system wbem set --enable true
When a host is rebooted and if 3rd party providers are present, the WBEM service will become enabled on reboot. In this instance the step of disabling WBEM services will have to be repeated.