Hi team!
Here's a case that Guest OS Suspension Caused by Disconnection Between ESX Hosts and Storage.
Symptom
Guest OS suspends because links between ESX hosts and storage are down.
1. RDM, VMFS3, VMFS5 are mapped to VMs on ESX hosts.
2. vdbench is used to deliver I/Os to raw LUNs and virtual disks.
3. Links between hosts and the storage are down.
4. VMs suspend and stop responding to users. The CPU usage is 100%.
5. VMs cannot run properly even after the links between hosts and storage are recovered.
Fault Diagnosis
The following introduces how VMFS metadata is renewed.
Before I/O processes access virtual disks, they need to obtain disk access permission and guarantee mutual exclusiveness using lock mechanism or atomic operations.
VMFS3 uses SCSI reserve mechanism while VMFS5 uses ATS atomic operations.
The flowcharts are as follows:
Lock process–SCSI reserve
ATS version (0x89)
During the previous process, when an I/O error occurs, VMFS will keep retrying (timeout surpassing system tolerance) until a file system error is generated.
This error causes VM exceptions such as VM suspension.
Log analysis:
A large quantity of logs are generated reporting the VMFS error.
When an error occurs in VMFS, VMs become abnormal.
Reference from VMware official website:
The knowledge base of VMware shows that if the storage performance is low, VMs will generate no responses and I/O suspends on both controllers of the array. However, this only happens in very extreme situations.
Solution
Restart the VM.
