Symptom:
On the VMware vCenter page, a disk in the storage device is displayed in grey, and its status is displayed Interrupted.
Diagnosis:
During the fault period, the logs on the storage device show that the LUN information is normal and the multipath query can be performed properly. At the same time, the logs on the ESX display that the paths of the LUN are added successfully, and the paths on the disk displayed in grey are in normal status. Therefore, the abnormal status of the disk is not caused by a fault on the storage system.
After further analysis of the logs on the storage device, it is discovered that the LUN was unmapped when VM services were running on it. After the LUN was unmapped from the storage system, a new LUN was mapped to the system. Then, the new LUN was unmapped, and the original LUN was mapped back to the storage system.
During the reproduction process, some problems are discovered. When the original LUN is unmapped from the storage system, because VM services are running on the LUN, the disk where the LUN resides has not been removed from the ESX. Instead, the ESX marks the disk in grey, attempting to recover the fault. At this time, all paths on the disk are displayed inactive. After a new LUN is mapped, it uses the host LUN ID of the original LUN. As a result, the two LUNs on the ESX use the same LUN ID.
The path information of each LUN on the ESX is identified by LUN ID. If two LUNs use the same LUN ID, the path identification is in disorder. One path on the original LUN is displayed as inactive. The other two paths on the new LUN are in active status. In fact, the three paths on the new LUN are normal. When the new LUN is unmapped and the original LUN is re-mapped, the scanning result on the VMware displays that the new LUN is successfully unmapped, but the original LUN is still displayed in grey. The two paths belong to the original LUN, and the path status is normal. However, the status of the original LUN is still displayed Interrupted. The information on the storage device shows that the original LUN and the three paths are normal.
In conclusion, this issue is caused by a disk fault that fails to be automatically recovered on the VMware due to misoperations. The original LUN is unmapped from the storage system when there are services running on the LUN. Due to the protection mechanism, the original LUN has not been removed from the VMware, which is marked inactive instead. Then, a new LUN on the storage device uses the ID of the original LUN. In this case, two LUNs on the VMware use the same ID, which causes the path management disorder on the VMware, and the fault fails to be recovered automatically.
Cause:
Due to misoperations, a LUN on which VM services are running is unmapped from the storage device. After the mapping and unmapping of a new LUN, the original LUN is finally re-mapped to the storage system. As a result, two LUNs on the VMware use the same LUN ID, which causes the path management disorder on the VMware, and the fault fails to be recovered automatically.
Solution:
Power off the target VM on which the LUN status is displayed in grey, including the VM to which disks are added.
Unmap the LUN from the storage system.
On vCenter, rescan the storage device and wait for a period of time (no more than 20 minutes) until the LUN is removed from the storage device.
Map the LUN to the storage device, and scan the storage device on vCenter. The result shows that the LUN exists in the storage device, and the LUN is in Connected status.
Check After Recovery:
After recovery, the VM can be powered on successfully.
