Some introduction about ATS which belonged to VAAI Block Primitives

VMware vSphere APIs – Array Integration (VAAI)

ATS Only Flag

An ATS only flag appears on newly created VMFS5 and VMFS6 datastores. When the host detects that the array supports ATS on the device, the ATS only flag is written to the datastore. From that point on, ATS always will be used on that datastore, so it can be used for all metadata operations.

The flag is not enabled, however, on VMFS5 volumes that were upgraded from VMFS3, facilitating seamless upgrades from VMFS3 to VMFS5. This is because VMFS3 was not ATS complete, and during the upgrade process, some hosts might continue to use SCSI reservations for certain metadata operations.

You cannot turn on the ATS only flag on VMFS5 volumes upgraded from VMFS3. The ATS only flag can be manually enabled or disabled on a newly created VMFS5. To manually enable it, you can use the hidden option:

# vmkfstools –configATSOnly 1 [device path]

Since there is no upgrade path from VMFS3 or VMFS5 to VMFS6, this is not a concern for VMFS6 volumes.

ATS Miscompare

When ATS was introduced for maintaining heartbeat “liveness” in vSphere 5.5U2, some issues arose. Some storage arrays returned ATS miscompares when the array was overloaded. In another scenario, reservations on the LUN on which the VMFS resided also caused ATS miscompares. In another case, the ATS “set” was correctly written to disk, but the array still returned a miscompare.

But it wasn’t all array related issues. Situations arose where VMFS also detected an ATS miscompare incorrectly. In this case, a heartbeat I/O (1) got timed-out and VMFS canceled that I/O, but before canceling the I/O, the I/O (ATS “set”) actually made it to the disk. VMFS next re-tried the ATS using the original “test-image” in step (1) since the previous one was canceled, and the assumption was that the ATS didn’t make it to the disk. Since the ATS “set” made it to the disk before the cancel, the ATS “test” meant that the in-memory and on-disk images no longer matched, so the array returned “ATS miscompare”.

When an ATS miscompare is received, all outstanding IO is canceled. This led to additional stress and load being placed on the storage arrays and degraded performance.

In vSphere 6.5, there are new heuristics added so that when an ATS miscompare event is encountered, VMFS reads the on-disk heartbeat data and verifies it against the ATS “test” and “set” images. This check to see if there is actually a real miscompare. If the miscompare is real, then we do the same as before, which is to cancel all the outstanding I/O. If the on-disk heartbeat data has not changed, then this is a false miscompare. In the event of a false miscompare, VMFS will not immediately cancled IOs. Instead, VMFS will now re-attempt the ATS heartbeat operation after a short interval (usually less than 100ms).

Thanks.