This post talks about the issue of too much time spent when expanding the disk capacity in the VMware VM. Please have a look at the information below.
[Issue description]
When adding one new disk (disk from the Huawei storage LUN) to the Virtual Machine, it only needs a few seconds. But when expanding this disk from 40GB to 4TB, it will need 4 minutes to finish.
The Vmware logs show that the Virtual Machine stopped during the capacity expansion:
2019-07-31T10:15:09.784Z| vcpu-0| I125: Checkpoint_Unstun: vm stopped for228266202 us
[Root cause]
VMware confirms that this issue is related with the vmfs6 datastor, and it will be fixed in the new version 6.5 u3 (current version is 6.5 u2).
6.5 u3 release notes:
https://docs.vmware.com/en/VMware-vSphere/6.5/rn/vsphere-esxi-65u3-release-notes.html
[Issue confirmed method]
1. After checking the LUN performance during the capacity expansion period from the storage side, we could find that VMware sent plenty of read operations when expanding the disk capacity, lasting for about 4 minutes (same with the time spent on the disk capacity expansion). There were very few write operations, concluding that the disk provisioning of Virtual Machine had the 'Thick Provision Lazy Zeroed' policy.
Also from the read and write latency, we can know there was no bottleneck from storage side.
2. Back to the read operation from Vmware, due to the default 'Sampling Interval of Historical Statistics Collection' is 60 seconds, so we modify it to 5 seconds then test the capacity expansion again, performance data shown as below. (Note that modify it back to 60 seconds after finishing the test).
We can find that during capacity expansion operation, Vmware sent continuously read operation whose speed is average 400MB/s to storage, after the read operation is done, the expansion also finishs.
(Note that following performance data comes from 40GB to 800GB capacity expansion, so the read operation only lasts for about 2 minutes).
Theoretically speaking, for Thick Provision Lazy Zeroed policy datastore, expanding capacity will only bring some metadata write operation to backend storage, and should not send so many read operations to backend storage.
3. Regarding this doubt, we did many tests in the lab, finally found this issue is related with vmfs6 datastore only, when using vmfs5 datastore, the expansion will finish soon under the same scenario.
At last, Vmware officially replied this issue is related with vmfs6 + Thick Provision Lazy Zeroed policy, shown as below.
This issue is also described in Vmware ESXi 6.5 u3 release notes:
https://docs.vmware.com/en/VMware-vSphere/6.5/rn/vsphere-esxi-65u3-release-notes.html
[Solution]
1. Use the vmfs5 datastore instead of vmfs6, or not to use Thick Provision Lazy Zeroed policy in the vmfs6 datastore.
2. Upgrade the VMware version to 6.5 u3.
(Note that Vmware ESXi 6.5 u3 currently has not finished the compatibility test with Huawei storage, so it is not recommended to use for now).
